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ABSTRACT:  The  primary  technical  objective  of  this  demonstration  project  was  to  evaluate  the  detection 
and  discrimination  capabilities  (including  production  rates  and  costs)  of  advanced  UXO  systems  in 
difficult  magnetic  clutter  environments  such  as  those  encountered  at  Kaho’olawe,  Hawaii.  One  90-m  by 
111.1-m  (1-hectare)  area  and  10  (not  necessarily  contiguous)  30-m  by  30-m  test  grids  within  the 
Kaho’olawe  Quality  Assurance  (QA)  Range  were  prepared  to  present  a  limited  range  of  target/clutter/ 
topography/vegetation/magnetic  background  conditions  to  the  various  demonstrators’  systems:  Geonics 
EM-63,  GTL  TM-5  EMU,  Geophex  GEM-3,  NRL  EMMS,  and  Geonics  EM-61.  Anomaly  maps,  survey 
maps,  and  demonstrators  target  discrimination  charts  are  compared  to  actual  groundtruth  to  determine 
performance  assessment  of  detection,  discrimination,  and  false  alarm  rate.  At  Kaho’olawe,  the  advanced 
EMI  systems  did  not  demonstrate  significant  performance  and/or  cost  improvements  over  the  baseline 
technology  consisting  of  a  standard  EM-6 1  system  operated  in  an  “EM  and  Flag”  mode.  This  was  not  true 
at  Jefferson  Proving  Ground,  Indiana  (July  2000).  Finally,  the  safety  and  logistics  problems  associated 
with  conducting  technology  demonstrations  concurrent  with  actual  UXO  cleanup  operations  proved  to  be 
a  very  inefficient,  costly,  and  time-consuming  process. 
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Preface 


The  work  documented  in  this  report  was  performed  during  the  period 
3  September  through  30  November  2001  as  part  of  the  Unexploded  Ordnance 
(UXO)  Technology  Demonstration  Program.  This  project  was  funded  by  the 
Department  of  Defense’s  Environmental  Security  Technology  Certification 
Program  (ESTCP)  directed  by  Dr.  Jeffrey  Marqusee  and  managed  by  Dr.  Anne 
Andrews,  UXO  Program  Manager.  Funding  for  the  analysis  documented  in  this 
report  was  provided  by  the  Installation  Restoration  Research  Program,  Project 
AF-25,  U.S.  Army  Corps  of  Engineers,  directed  by  Dr.  M.  John  Cullinane, 
Environmental  Laboratory  (EL),  U.S.  Army  Engineer  Research  and 
Development  Center  (ERDC),  Vicksburg,  MS. 

Mr.  Hien  Dinh,  Naval  Explosive  Ordnance  Disposal  Technology  Division, 
Indian  Head,  Maryland,  had  overall  responsibility  for  the  ESTCP  project,  was 
responsible  for  procuring  and  preparing  the  UXO  test  targets  and  emplaced 
clutter  items  used  for  this  study,  participated  in  the  field  demonstration  activities, 
and  was  responsible  for  postdemonstration  excavation  and  groundtruth  verifica¬ 
tion  activities.  Dr.  Ernesto  R.  Cespedes,  EL,  ERDC,  was  responsible  for  planning 
and  coordinating  the  field  demonstrations.  He  developed  the  metrics  used  to 
evaluate  the  performance  of  the  demonstrators,  participated  in  the  site  prepara¬ 
tion  and  field  demonstration  activities,  and  directed  the  analysis  of  the  results. 

Mr.  George  Robitaille,  U.S.  Army  Environmental  Center,  was  responsible  for 
providing  UXO  targets  from  the  UXO  Standardized  Test  Site  project  inventory. 

This  project  was  performed  under  the  general  supervision  of  Dr.  David 
Tazik,  Chief,  Ecosystems  Evaluation  and  Engineering  Division,  and 
Dr.  Elizabeth  C.  Fleming,  Acting  Director,  EL.  Commander  and  Executive 
Director  of  ERDC  was  COL  James  R.  Rowan,  EN.  Director  was  Dr.  James  R. 
Houston. 
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Introduction 


The  Environmental  Security  Technology  Certification  Program  (ESTCP) 
funded  the  Naval  Explosive  Ordnance  Disposal  Technology  Division 
(NAVEODTECHDIV)  (lead  agency),  the  U.S.  Army  Engineer  Research  and 
Development  Center  (ERDC),  and  the  U.S.  Army  Environmental  Center  (AEC) 
to  design  and  conduct  controlled  demonstrations  of  advanced  unexploded 
ordnance  (UXO)  detection  and  discrimination  technologies  (Appendix  A).  The 
demonstrations  were  conducted  at  two  prepared  sites  located  on  the  island  of 
Kaho’olawe  in  Hawaii  (Figure  1)  during  the  period  3  September  through 
30  November  2001  and  were  designed  to  evaluate  the  capabilities  of  state-of-the- 
art  technologies  to  detect,  discriminate,  and  identify  buried  UXO  in  areas  con¬ 
taining  high  concentrations  of  natural  (magnetic  rocks/soils)  and  man-made 
(munitions  fragments)  clutter.  This  report  documents  the  results  of  these  demon¬ 
strations  and  provides  data  to  aid  the  Government  in  selecting  effective  and 
efficient  systems  for  UXO  detection  and  discrimination  in  difficult  magnetic  sites 
such  as  those  encountered  at  Kaho’olawe  Island. 


Background  Information 

The  Department  of  Defense  (DoD)  is  currently  involved  in  a  number  of  UXO 
site  remediation  efforts  where  rapid  transition  of  advanced  technologies  can 
potentially  improve  UXO  detection  efficiency,  save  substantial  sums  of  money 
by  reducing  false  alarms,  and  significantly  expedite  the  transfer  of  lands  for  re¬ 
use.  One  of  the  most  prominent  of  these  efforts  is  the  ongoing  UXO  cleanup  of 
the  Kaho’olawe  bombing  ranges.  The  major  difficulty  with  this  site  is  that  the 
significant  magnetic  anomalies  from  geologic  sources  and  near-surface  metal 
fragments  make  traditional  magnetometer-based  surveys  impractical.  Active 
electromagnetic  induction  (EMI)  sensors  such  as  the  Geonics  EM-61  and  the 
Geographical  Technology  Limited  (GTL)  TM-5  EMU  are  currently  the  primary 
sensors  being  used  by  the  contractors  at  Kaho’olawe.  Even  though  these  EMI 
sensors  have  proven  more  effective  at  this  site  than  passive  magnetometers,  their 
detection  performance  at  Kaho’olawe  sites  has  not  been  quantified,  and  they  have 
been  subject  to  very  high  false  alarm  rates.  Parsons-UXB,  the  prime  UXO 
contractor  at  Kaho’olawe,  reports  that  as  of  14  November  2001  they  have 
detected  61,261  subsurface  anomalies  and,  after  digging,  they  have  found  that 
only  2.7  percent  are  UXO,  27  percent  are  false  positives  from  geologic  sources, 
and  70.3  percent  are  the  result  of  buried  metal  from  both  UXO  and  non-UXO- 
related  materials.  It  should  be  noted  that  it  is  not  possible  to  evaluate  the  detec¬ 
tion  performance  (Probability  of  Detection  or  Pd)  from  these  findings,  since  the 
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actual  number  of  buried  UXO  (groundtruth)  is  not  known.  ESTCP  funded  this 
project  to  address  the  critical  need  for  more  effective  and  efficient  UXO 
technologies  at  sites  such  as  Kaho’olawe. 

The  first  phase  of  this  ESTCP  project  was  conducted  at  Jefferson  Proving 
Ground  (JPG),  Indiana,  during  the  period  July  through  November  2000.  This 
phase  involved  three  advanced  EMI  sensing  system  demonstrators:  (a)  North 
American  Exploration  of  Virginia,  Inc.  (NAEVA),  Geophysics  employing  the 
Geonics  EM-63  multi-channel,  time-domain  EMI  system,  (b)  the  Naval  Research 
Lab  (NRL)  employing  the  single-channel,  time-domain  Electromagnetic 
Manportable  System  (EMMS),  and  (c)  Geophex  Ltd.  employing  the  multi¬ 
frequency,  frequency-domain  GEM-3  system.  In  addition,  a  commercial  UXO 
surveying  firm.  Explosive  Ordnance  Disposal  Technology,  Inc.  (EODT),  was 
contracted  to  conduct  standard  “mag  and  flag”  surveys  of  the  JPG  test  areas  in 
order  to  compare  the  EMI  systems’  performance  with  conventional  techniques. 
The  results  of  this  first  phase  of  demonstrations  are  documented  by  Cespedes 
(2001)  and  indicate  that  the  EMI  systems  perform  considerably  better  (higher 
detection  rates,  lower  false  alarms)  than  standard  “mag  and  flag”  surveys, 
especially  in  areas  containing  high  levels  of  magnetic  clutter  from  geologic 
sources.  Since  the  first  phase  results  provided  strong  indications  that  these 
technologies  can  significantly  reduce  false  alarms  resulting  from  high  magnetic 
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permeability  in  the  soils  and  roeks,  it  was  decided  to  evaluate  the  three  systems 
under  the  more  realistic  and  difficult  geologic  conditions  found  in  Kaho’olawe. 

In  addition  to  the  three  advanced  EMI  systems  demonstrated  during  the  first 
phase,  the  ESTCP  Program  Office  agreed  to  allow  Geophysical  Technology 
Limited  (GTL)  to  demonstrate  its  advanced  EMI  sensor  system,  the  TM-5 
Electromagnetic  Unit  (EMU),  as  part  of  the  second  phase  tests  at  Kaho’olawe. 
GTL  provided  its  own  funding  to  participate  in  this  phase,  and  ESTCP  agreed  to 
fund  the  additional  costs  associated  with  monitoring  the  GTL  field  surveys  and 
evaluating  their  performance.  It  should  be  noted  that  GTL  has  participated  in 
previous  demonstrations  conducted  at  JPG  and  was  among  the  top  performers  in 
several  of  the  test  scenarios  including  the  small  UXO  sites  (e.g.,  grenades  and 
submunitions)  (U.S.  Army  Environmental  Center  (USAEC)  and 
NAVEODTECHDIV  1997).  In  addition,  GTL  has  considerable  operational 
experience  with  the  TM-5  EMU  in  Kaho’olawe  live  sites. 

For  baseline  comparisons  with  technologies  currently  used  at  Kaho’olawe 
sites,  Parsons-UXB  conducted  standard  EM-61  digital  surveys  as  well  as  “EM 
and  Flag”  surveys  using  the  EM-61  and  the  TM-5  EMU  in  a  real-time 
detection/discrimination  mode. 

The  focus  of  this  demonstration  project  was  to  evaluate  these  advanced  EMI 
technologies  under  realistic  and  difficult  field  conditions  in  order  to  quantify 
their  detection,  discrimination,  cost,  and  production  rates  while  operating  at 
several  areas  within  Kaho’olawe  with  varying  degrees  of  target/clutter  densities 
and  magnetic  noise  levels.  The  purpose  of  this  report  is  to  aid  managers  of  UXO 
cleanup  projects  as  well  as  regulators  and  other  stakeholders  to  make  informed 
decisions  concerning  the  capabilities,  costs,  and  risks  associated  with  applying 
these  technologies  to  their  site-specific  UXO  remediation  problems. 


Official  DoD  Requirement  Statement(s) 

This  project  addresses  the  Tri-Service  Environmental  Quality  Research, 
Development,  Test,  and  Evaluation  Strategic  Plan,  UXO  requirements,  and  more 
specifically,  the  U.S.  Army  requirement  A(1.6a),  titled:  Unexploded  Ordnance 
(UXO)  Screening,  Detection,  and  Discrimination.  Additionally,  the  project 
describes  the  FY99  Army  Environmental  Requirements  and  Technology 
Assessments  (AERTA).  This  Army  requirement  has  been  ranked  as  the  highest 
priority  user  need  in  the  Environmental  Cleanup  Pillar.  In  addition,  this  project 
addresses  the  UXO  detection  and  discrimination  requirements  and  recommenda¬ 
tions  described  in  the  Defense  Science  Board  Task  Force  Final  Report  on  UXO 
Clearance  and  Remediation  published  in  1998  and  will  provide  data  to  support 
the  development  of  more  accurate  estimates  of  the  overall  DoD  UXO  environ¬ 
mental  remediation  costs. 

The  advanced  technologies  demonstrated  as  part  of  this  effort  address  all 
aspects  of  the  requirements  for  land-based,  man-portable  buried  UXO  detection 
and  discrimination  systems.  The  results  of  these  demonstrations  will  be  used  to 
quantify  the  capability  of  state-of-the-art  systems  to  detect,  discriminate,  locate. 
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and  identify  buried  targets.  The  performance  of  the  advanced  systems  was 
compared  with  the  baseline  capability  demonstrated  by  the  onsite  contractor, 
Parsons-UXB. 

This  technology  demonstration  creates  a  framework  for  the  evaluation  of 
state-of-the  art  sensor  technologies  to  detect,  locate,  and  identify  UXO.  Baseline 
technology  performance  is  established,  and  technology  capabilities  and  limita¬ 
tions  are  assessed.  Results  from  this  program  will  be  widely  distributed  to  aid  in 
the  selection  and  utilization  of  sensors  and  data  analysis  techniques  for  UXO 
characterization  and  restoration  efforts. 


Objectives  of  the  Demonstration 

The  primary  technical  objective  of  this  demonstration  project  was  to  evaluate 
the  detection  and  discrimination  capabilities  (including  production  rates  and 
costs)  of  advanced  UXO  systems  in  difficult  magnetic  clutter  environments  such 
as  those  encountered  at  Kaho’olawe.  For  this  phase  of  the  project,  one  90-m  by 
1 1 1.1-m  (1-hectare)  area  and  10  (not  necessarily  contiguous)  30-m  by  30-m  test 
grids  within  the  Kaho’olawe  Quality  Assurance  (QA)  Range  were  prepared  to 
present  a  limited  range  of  target/clutter/topography/vegetation/magnetic  back¬ 
ground  conditions  to  the  various  demonstrators. 

The  evaluation  objectives  for  this  phase  of  demonstrations  were  as  follows: 

a.  To  evaluate  the  demonstrators’  detection  and  discrimination  capabilities 
by  means  of  surveys  of  ten  30-m  by  30-m  grids  and  one  1 -hectare  area 
within  the  Kaho’olawe  QA  Range  under  realistic  target/  geologic  clutter/ 
man-made  clutter/topography  scenarios  while  operating  as  efficiently  as 
possible  (minimizing  time,  manpower,  and  costs). 

b.  To  evaluate  the  demonstrators’  ability  to  analyze  survey  data  in  a  timely 
manner  and  provide  prioritized  “dig  lists”  with  associated  confidence 
levels. 

c.  To  collect  data  on  manpower  and  time  required  to  collect  field  data 
necessary  to  produce  their  final  products  (prioritized  dig  sheets  and 
georeferenced  anomaly  maps). 

d.  To  compare  the  performance  of  the  advanced  systems  with  the  baseline 
technologies  that  are  currently  employed  at  Kaho’olawe. 

e.  To  provide  high  quality,  well  ground-truthed,  georeferenced  data  for 
postdemonstration  analysis  and  development  of  Receiver  Operating 
Characteristic  (ROC)  curves. 


Regulatory  Issues 

The  principal  regulatory  issue  affecting  UXO  detection  and  discrimination 
technologies  is  gaining  confidence  and  approval  from  Federal,  state,  and  local 
regulators,  stakeholders,  and  users.  In  addition,  acceptance  of  these  innovative 
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technologies  from  agencies  such  as  the  U.S.  Army  Corps  of  Engineers  and  the 
Naval  Facilities  and  Engineering  Command  is  needed  to  ensure  that  future 
Requests  for  Proposals  (RFPs)  for  UXO  cleanup  projects  will  be  written  in  a 
manner  that  will  either  sanction  these  technologies,  or  at  least  allow  their 
inclusion  in  proposals  for  site  work.  Members  of  the  regulatory  community  who 
are  aware  of  these  technology  demonstrations  are  listed  in  Appendix  A. 


Previous  Testing  of  the  Technology 

Versions  of  the  technologies  demonstrated  under  this  effort  have  been 
previously  tested  as  part  of  other  DoD  and  Army  sponsored  demonstrations 
including  the  Defense  Advanced  Research  Projects  Agency  (DARPA)  Clutter 
Experiment  (FY97),  the  Jefferson  Proving  Ground  Phases  II  through  IV 
Demonstrations,  and  a  number  of  ESTCP-fiinded  field  demonstration  projects. 
However,  this  ESTCP  project  represents  the  first  set  of  controlled  field  experi¬ 
ments  at  an  actual  remediation  site  where  these  advanced  technologies  have  been 
tested  under  realistic  conditions  that  allowed  for  side-by-side  comparison  of 
detection/discrimination  performance,  production  rates,  and  costs. 
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Description 

The  five  eleetromagnetic  induction  sensing  systems  that  participated  in  this 
demonstration  project  at  Kaho’olawe  included  the  following  (in  chronological 
order): 

a.  Geonics  Ltd.  EM-63,  a  multichannel  time  domain  EMI  sensor  operated 
by  personnel  from  NAEVA  Geophysics 

b.  GTL  TM-5  EMU,  a  multiperiod  time-domain  EMI  sensor  operated  by 
GTL  and  Parsons-UXB  Technology  personnel 

c.  Geophex  Ltd.  GEM-3,  a  multichannel  frequency-domain  EMI  sensor 
system  operated  by  Geophex  Ltd.  Personnel 

d.  NRL  Man-Portable  EM  System  (EMMS)  adjunct  to  the  MTADS  system, 
a  single-channel  time-domain  EMI  sensor  operated  by  personnel  from 
NRL  with  processing  support  from  AETC  Corp. 

e.  Geonics  EM-6 1 ,  a  single  channel  time  domain  EMI  system  operated  by 
Parsons-UXB. 

Each  of  the  five  sensors  was  integrated  into  a  man-portable  platform  that 
included  data  acquisition/storage  that  merged  the  sensor  data  with  position  data 
collected  by  differential  Global  Positioning  System  (GPS)  receivers.  In  addition. 
Parsons  Technology  operated  the  TM-5  EMU  and  the  EM-61  in  a  field  discrimi¬ 
nation  (“EM  and  Flag”)  mode.  In  this  mode,  the  systems  did  not  record  digital 
sensor  data,  and  the  only  permanent  record  consists  of  the  identified  UXO  loca¬ 
tions  that  were  marked  by  a  separate  GPS  survey  crew.  These  sensor  systems  are 
shown  in  Figures  2  through  7  conducting  surveys  at  the  Kaho’olawe  QA  range. 
Detailed  descriptions  of  these  sensing  systems  are  included  in  studies  by  Geonics 
Limited  (1999  and  2000),  GTL  (2002),  Won  et  al.  (1998),  and  NRL  (2001). 

Strengths,  Advantages,  and  Weaknesses 

The  following  paragraphs  represent  a  summary  of  the  perceived,  claimed, 
and  documented  capabilities  of  each  of  the  sensors  employed  by  the  technology 
demonstrators. 
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Figure  2.  EM-63  operated  by  NAEVA  Figure  3.  TM-5  EMU  operated  by  GTL 


Figure  4.  GEM-3  operated  by  Geophex,  Ltd.  Figure  5.  EMMS  operated  by  NRL 


Figure  6.  EM-61  operated  by  Parsons  Figure  7.  TM-5  EMU  operated  by  Parsons 


Geonics  EM-63 


The  claimed  strengths  of  the  EM-63  lie  in  its  ability  to  rapidly  collect 
multiple  channels  of  time-domain  information  at  each  survey  point.  The  EM-63 
collects  up  to  26  geometrically  spaced  time  gates  covering  the  time  decay 
response  in  the  range  from  1 80  |J,sec  to  25  msec  after  pulsing  of  the  transmitter 
coil  (Geonics  Limited  1 999).  Since  the  shape  of  the  decay  curve  is  dependent  on 
the  shape,  size,  orientation,  and  metal  composition  of  the  buried  object,  it  is 
hypothesized  that  the  EM-63  data  channels  can  provide  the  information  required 
to  detect,  discriminate,  and  identity  the  buried  UXO  targets,  and  to  significantly 
reject  responses  from  geologic  materials  and  metallic  clutter.  The  EM-63  is  a 
commercially  available  sensor  (produced  by  Geonics  Ltd.  which  also  manu¬ 
factures  the  EM-61)  and  has  been  ruggedized  for  field  use.  Another  significant 
strength  of  the  system  demonstrated  at  JPG  is  the  processing  expertise  of 
NAEVA  personnel.  During  previous  JPG  demonstrations,  NAEVA  has  con¬ 
sistently  ranked  among  the  top  performers,  even  though  they  had  employed 
sensor  data  like  other  less-successful  demonstrators.  Pereeived  weakness  of  the 
NAEVA  EM-63  demonstration  system  was  that  NAEVA  had  limited  experienee 
with  the  EM-63,  since  it  has  only  recently  become  available  and  has  undergone  a 
number  of  hardware  and  software  modifications.  In  addition,  the  analysis 
techniques  were  still  under  development  and  had  not  been  fully  tested  in  difficult 
conditions  such  as  those  encountered  in  Kaho’olawe. 


GTL  TM-5  EMU 

The  GTL  TM-5  EMU  is  a  multiperiod,  time-domain  EMI  system  integrated 
with  real-time  processing  that  is  claimed  to  provide  automatic  background 
leveling  capability  for  enhanced  detection  and  discrimination  capabilities  in  sites 
containing  high  levels  of  magnetic  interference  (GTL  2002).  Unlike  the  other 
EMI  systems  tested  during  this  demonstration,  the  TM-5  is  a  monocoil  sensor 
with  one  element  acting  as  both  transmitter  and  receiver.  The  TM-5’s  transmitted 
waveform  is  referred  to  as  “multiperiod”  because  it  consists  of  a  wavetrain  with  a 
single,  longer  pulse  followed  by  three  shorter  pulses  with  the  same  length,  all  of 
which  are  repeated  at  a  rate  of  approximately  1,200  Hz.  The  long  pulse  is  four 
times  wider  than  each  of  the  short  pulses.  Each  decay  period  following  each  of 
the  four  pulses  is  sampled  twice,  with  the  specific  details  of  gate  timing,  periods, 
and  the  method  of  combining  them  for  analysis  being  proprietary  information  of 
MineLab  Electronics  and  not  available  for  publication.  GTL  has  taken  the  sensor 
and  electronics  provided  by  MineLab  and  integrated  advanced  processing  and 
positioning  information  to  produce  the  TM-5  EMU  specifically  for  UXO  detec¬ 
tion  applications.  The  TM-5  EMU  can  perform  UXO  discrimination  in  real  time, 
or  the  data  can  be  recorded  in  digital  form  and  postprocessed  to  allow  for  more 
in-depth  analysis  and  interpretation.  Unfortunately,  descriptions  of  the  real-time 
and  postprocessing  techniques  are  also  proprietary  to  GTL  and  not  releasable.  In 
spite  of  GTL’s  assurance  that  these  details  would  be  fully  disclosed  in  return  for 
the  Govermnent’s  agreeing  to  allow  their  participation  in  the  Kaho’olawe 
demonstrations,  such  information  has  not  been  provided.  Thus,  claims  as  to  the 
TM-5  EMU’s  automatic  background  leveling  capabilities  and  its  superior  ability 
to  operate  in  highly  conductive  and/or  magnetic  environments  cannot  be  fully 
evaluated. 
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Geophex  GEM-3 

The  strength  of  the  GEM-3  system  is  claimed  to  lie  in  its  ability  to  rapidly 
collect  multiple  channels  of  complex  frequency  domain  EMI  data  over  a  wide 
range  of  audio  frequencies  (30  Hz  to  over  20  kHz).  This  allows  for  performing 
what  Geophex  Ltd.,  the  developer  of  the  system,  calls  Electromagnetic  Induction 
Spectroscopy  (EMIS)  on  buried  objects  (Won  et  al.  1998).  EMIS  provides  a 
method  to  discriminate  UXO  targets  from  natural  and  manmade  clutter  objects 
by  means  of  their  unique,  complex  (inphase  and  quadrature)  frequency  responses. 
The  GEM-3  system  was  the  top  performer  in  the  discrimination  and  identifi¬ 
cation  tests  conducted  during  JPG  Phase  IV.  A  concern  and  possible  weakness  of 
the  GEM-3  is  that,  to  accomplish  a  wide-area  detection  and  discrimination  survey 
in  a  reasonable  time,  it  must  keep  the  number  of  frequencies  to  a  relatively  low 
number  (seven  in  the  case  of  these  Kaho’olawe  tests)  and  must  transmit  them 
simultaneously  rather  than  sequentially  as  was  done  during  the  static  JPG  IV 
tests.  This  simultaneous  transmission  of  multiple  frequencies  may  reduce  the 
power  dedicated  to  each  frequency,  and  this  could  affect  the  depth  capability  of 
the  GEM-3  system.  Another  possible  weakness  of  the  GEM-3  system  is  that 
high-accuracy  position  information  is  required  to  perform  the  discrimination.  In 
the  previous  JPG  IV  demonstrations,  this  high  degree  of  position  accuracy  was 
obtained  by  means  of  templates  placed  over  specified  target  locations  allowing 
static  point  measurements  to  be  made.  That  approach  was  not  viable  for  the  wide- 
area  search  requirements  of  the  current  project,  and  Geophex  had  to  rely  on  GPS 
position  information,  which  results  in  significantly  greater  position  errors  and 
sparser  data  sets.  Finally,  it  was  observed  that  the  GEM-3  system  is  still  under¬ 
going  development,  and  the  sensor  design,  platform,  data  acquisition  system,  and 
analysis  approaches  have  not  been  finalized  or  optimized. 


NRL  EMMS 

The  EMMS  is  derived  from  the  highly  successful  Multi-Sensor  Towed  Array 
Detection  System  (MTADS)  development  effort  and  thus  incorporates  many  of 
its  sensing,  navigation,  and  data  analysis  system  (DAS)  advances  demonstrated 
and  documented  in  a  number  of  ESTCP-funded  field  demonstrations.  The  speci¬ 
fications  and  performance  improvements  incorporated  into  the  version  of  the 
EMMS  demonstrated  at  Kaho’olawe  are  fully  described  in  the  ESTCP  report 
titled  “Man-Portable  Adjuncts  for  the  MTADS”  (NRL  2001).  Based  on  a  modi¬ 
fied  version  of  the  commercially  available  Geonics  EM-61  (with  the  0.5-m  by 
1.0-m  transmitter  coil),  the  most  widely  used  EMI  system  for  UXO  detection 
applications,  the  EMMS  sensor  is  expected  to  have  good  UXO  detection  capa¬ 
bility  to  the  maximum  depths  of  the  objects  emplaced  at  Kaho’olawe.  Coupled 
with  the  very  high  accuracy  of  the  MTADS-derived,  digital  inclinometer/GPS 
system,  the  EMMS  is  expected  to  produce  high  quality  georeferenced  EMI  data. 
A  potential  limitation  of  the  EMMS  is  the  single  channel  of  data  available,  which 
may  limit  the  discrimination  performance  compared  to  what  can  ultimately  be 
achieved  by  multichannel  systems. 
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Geonics  EM-61 


The  Geonics  EM-61  system  is  a  single-channel  time-domain  metal  detector. 
It  is  the  most  mature  and  widely  applied  EMI  technology  for  UXO  detection 
surveys  (Geonics  Limited).  The  system  is  available  with  different  coil  configura¬ 
tions  and  the  one  used  at  Kaho’olawe  by  Parsons-UXB  incorporated  the  large 
1-m  by  1-m  coils.  As  is  the  case  with  other  time-domain  systems,  the  transmitter 
coil  generates  a  pulsed  primary  magnetic  field  in  the  earth,  which  induces  eddy 
currents  in  nearby  metallic  objects.  The  eddy  current  decay  produces  a  secondary 
magnetic  field  measured  by  the  receiver  coil.  By  taking  the  measurement  at  a 
relatively  long  time  after  the  start  of  the  decay,  the  designers  of  the  EM-61  pre¬ 
dict  that  the  currents  induced  in  the  ground  have  fully  dissipated  and  only  the 
current  in  buried  metal  objects  is  still  producing  a  secondary  field.  The  responses 
are  recorded  and  displayed  by  an  integrated  data  logger.  Another  advantage  of 
the  EM-61  system’s  reputation  as  the  EMI  system  of  choice  at  UXO  sites  is  that 
it  is  supported  by  a  number  of  software  development  firms.  For  example, 
Geosoft,  one  of  the  leading  designers  of  software  for  geophysical  applications, 
produces  a  version  of  its  UX-Detect  software  specifically  tailored  to  accept  and 
analyze  EM-61  data.  Similar  to  the  EMMS  described  above,  the  primary  per¬ 
ceived  weakness  of  the  EM-61  system  is  that  its  single-channel  time  decay 
window  does  not  provide  adequate  information  to  discriminate  buried  UXO  from 
other  conductive  or  highly  permeable  buried  objects  (both  manmade  and 
naturally  occurring). 


Factors  Influencing  Cost  and  Performance 

Data  on  the  various  factors  that  influence  the  overall  cost  and  performance  of 
each  of  these  systems  in  actual  UXO  remediation  efforts  were  collected  as  part  of 
this  field  demonstration  effort,  and  include  the  following: 

a.  Equipment  setup,  calibration  time,  and  man-hour  requirements. 

b.  Time  and  man-hour  requirements  to  survey  the  demonstration  test  areas. 

c.  Downtime  resulting  from  system  malfunctions  and  maintenance 
requirements. 

d.  Reacquisition/resurvey  time  and  man-hour  requirements. 

e.  Accuracy  of  georeferenced  maps  and  prioritized  dig  lists  with  respect  to: 

( 1 )  Probability  of  Detection  (P(det)) . 

(2)  False  alarm  rates  (Pfp,  FAR,  Total  FAR). 

(3)  Discrimination  capability  (P(disc)). 

(4)  Identification  capability. 

(5)  Target  location  accuracy. 

The  Demonstration  Workplan  (Naval  Explosive  Ordnance  Disposal 
Technology  Division  (NAVEODTECHDIV)  2001a)  and  Chapter  4  of  this  report 
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include  detailed  deseriptions  of  the  methods  and  metrics  used  to  evaluate  each  of 
the  eost  and  performance  factors. 
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3  Site/Facility  Description 


Background 

Site  selection  criteria,  site  description,  and  site  preparation  activities  are 
described  in  detail  in  the  Site  Preparation  Plan  (NAVEODTECHDIV  2001b),  but 
short  descriptions  of  the  island,  the  calibration  and  demonstration  sites,  and  the 
emplaced  targets  are  presented  in  this  section. 

Kaho’olawe  Island  consists  of  the  summit  of  a  single  volcanic  dome  that 
reaches  a  peak  elevation  of  450  m  (1,477  ft)  above  mean  sea  level  (Lua  Makika 
point  at  the  northeastern  part  of  the  island).  It  is  one  of  the  oldest  of  the  main 
group  of  Hawaiian  Islands  and  is  separated  from  Maui  by  the  1 1-km  (6.9-mile) 
wide  Alalake’ike  Channel  and  from  Lana’i  by  the  28  km  (17.5  mile) 

Kealaikahiki  Channel.  It  is  18  k  (1 1  miles)  long  and  10  km  (6  miles)  wide  with 
an  area  of  28,776  acres.  Kaho’olawe  Island  was  used  as  a  weapons  range  and 
military  training  area  from  1941  until  1990.  Title  X  of  the  FYl  994  Department  of 
Defense  Appropriations  Act  was  enacted  in  November  1993  and  directed  the 
cleanup  of  ranges  in  Kaho’olawe  Island.  Title  X  allocated  $400  M  for  UXO 
remediation  starting  in  1993  and  required  that  Kaho’olawe  be  transferred  to  a 
Native  Hawaiian  sovereign  entity  no  later  than  November  2003. 

The  island’s  geology  consists  predominantly  of  basalt,  hardpan,  and  sand. 

The  magnetite-containing  basaltic  rocks  and  soils  have  precluded  the  use  of 
magnetometers  for  UXO  detection  and  have  been  the  source  of  a  significant 
number  of  false  alarms  encountered  by  the  EMI  sensors  currently  used  by  the 
UXO  remediation  contractors.  Kaho’olawe’s  surface  features  consist  primarily  of 
dry  land  vegetation  and  hardpan  (Figures  8  and  9).  The  island’s  climate  is  windy 
and  very  dry,  averaging  only  25.4  to  50.8  cm  (10  to  20  in.)  of  rainfall  per  year 
(mostly  on  the  eastern  side  of  the  island).  The  island  has  had  a  history  of  over- 
grazing  by  sheep  and  cattle,  destruction  of  vegetation  by  goats,  deforestation  by 
settlers,  and  finally,  damage  caused  by  target  bombing  and  shelling  by  the 
United  States  Military. 


Site/Faciiity  Characteristics 

The  previously  established  Kaho’olawe  UXO  Remediation  Project’s  Quality 
Control  (QC)  and  Quality  Assurance  (QA)  ranges,  shown  in  Figure  10,  were 
selected  for  this  project’s  calibration  (Figure  11)  and  demonstration  areas, 
respectively.  The  calibration  and  demonstration  sites  were  surveyed  using 
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Figure  8.  Overview  of  Kaho’olawe  showing  erosion  of  terrain 


Figure  9.  Overview  of  Kaho’olawe  vegetation  and  hardpan  terrain 

electronic  theodolites  (Leica  Model  TCA  1 102)  and  real-time  kinematic  differ¬ 
ential  global  positioning  system  (RTK-DGPS)  survey  equipment.  Both  sites  were 
oriented  to  true  north  and  each  comer  of  each  range  was  staked  with  a  ferrous  rod 
and  its  coordinates  recorded.  The  magnetic  variation  at  the  Kaho’olawe  site  is 
9°-59’  East.  Monuments  near  the  calibration  and  demonstration  areas  were 
brought  up  to  first-order  accuracy  during  the  initial  site  preparation  activities,  and 
updated  coordinates  were  provided  to  the  demonstrators  prior  to  the  scheduled 
demonstrations. 
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Calibration  area 


The  calibration  area  consisted  of  three  30-m  by  30-ni  grids  and  was  estab¬ 
lished  to  allow  the  demonstrators  to  conduct  system  calibration,  signature 
collection,  and  algorithm  development  prior  to  participating  in  the  blind  tests 
conducted  in  the  demonstration  areas.  The  complete  groundtruth  of  all  items 
emplaced  in  the  calibration  area  was  made  available  to  each  demonstrator  prior  to 
arrival  on  Kaho’olawe. 

The  demonstration  area  was  subdivided  into  three  areas:  Area  A  consisted  of 
four  30-m  by  30-m  grids.  Area  B  consisted  of  nine  30-m  by  30-m  grids  and  three 
partial  grids  (totaling  1  hectare),  and  Area  C  consisted  of  six  30-m  by  30-m  grids 
(Figure  12).  Figure  13  shows  the  QA  range  looking  from  grid  5B  to  grid  IE. 


1  2  3  4  5  6 


Figure  12.  Layout  of  demonstration  areas 
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Target  emplacement 

The  target  list  eonsisted  of  the  following  18  items  (items  in  bold  were  taken 
from  the  Standard  Test  Sites  Program  stoekpiie  and  the  rest  are  from  Kaho’olawe 
UXO  cleanup  operations): 

•  20-milIimetcr  (mm)  projectile 

•  40-mm  projectile 

•  60-mm  mortar 

•  8 1  -mm  mortar 

•  2.25-in.  rocket 

•  2.75-in.  rocket  warhead 

•  3-in.  projectile 

•  5-in.  projectile 

•  105-mm  projectile 

•  SMAW  rocket 

•  LAAW 
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•  BDU-3  practice  bomb 

•  BDU-33  practice  bomb 

•  MK-82  practiee  bomb 

•  MK-3  practice  bomb 

•  MK-81  practice  bomb 

•  MK- 106  practice  bomb 

•  MK-83  practice  bomb 

All  unfired,  inert  UXO  items  were  thoroughly  degaussed  either  at 
NAVEODTECHDIV,  at  Aberdeen  Proving  Ground,  Maryland,  or  in  the  field 
prior  to  emplacement  in  order  to  remove  any  remnant  magnetic  moment.  Each 
degaussed  item  was  checked  for  remnant  magnetization  using  a  G-858  total  field 
magnetometer.  Target  and  clutter  items  were  weighed,  measured,  and  photo¬ 
graphed,  and  the  excavation  crew  emplaced  the  preselected  target/clutter  item  at 
the  location,  orientation,  and  approximate  burial  depth  specified  in  the  Site 
Preparation  Plan. 

Prior  to  emplacing  any  targets,  all  UXO  target  emplacement  locations  (larger 
than  20  mm)  were  surveyed  with  a  Geonics  EM-61  HH  and  a  Geometries  G-858 
in  order  to  record  background  noise  prior  to  emplacement  and  also  to  allow 
removal  of  any  metal  objects  in  the  vicinity  (within  a  2-m  by  2-m  area)  of  the 
target  location.  In  addition,  one  30-m  by  30-m  grid  within  the  Demonstration 
area  that  contained  no  preemplaced  items  was  surveyed  with  both  the  EM-61  and 
the  G-858.  These  preemplacement  surveys  are  included  in  the  data  archive 
described  in  Appendix  B.  Figures  14  and  15  show  overviews  of  the  clutter  and 
target  items  in  the  calibration  and  demonstration  areas,  respectively.  Detailed 
information  on  these  items  is  included  in  the  Site  Preparation  Plan. 
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Figure  14.  Kaho’olawe  calibration  area  target  emplacement  plan 
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Figure  15.  Kaho’olawe  site  map  showing  emplaced  items  in  Areas  A,  B,  and  C  of  the  demonstration  site 
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Performance  Objectives 

The  objective  of  this  demonstration  was  to  evaluate  the  performance  of 
advanced  EMI  technologies  in  very  difficult  UXO  target,  clutter,  and  geologic 
noise  scenarios  such  as  those  commonly  encountered  in  Kaho’olawe  UXO 
remediation  efforts.  As  was  the  case  at  JPG  during  the  first  phase  of  this  project, 
this  demonstration  attempted  to  evaluate  the  detection,  discrimination,  and 
identification  capabilities  of  advanced  UXO  systems  while  simultaneously 
evaluating  their  production  rates,  manpower  requirements,  and  costs. 

The  scope  of  this  effort  was  to  collect  sufficient  information  from  a  limited 
range  of  test  scenarios  in  order  to  quantify  the  advantages  and  disadvantages  of 
each  of  the  four  EMI  technologies  so  that  they  may  be  properly  applied  to  spe¬ 
cific  UXO  cleanup  problems.  The  immediate  goal  of  this  effort  is  to  quantify  any 
performance  and  cost  improvements  resulting  from  the  application  of  advanced 
EMI  technologies  so  that  they  can  be  rapidly  transitioned  to  Kaho’olawe-type 
operations  where  natural  (magnetic  rocks/soils)  and  man-made  (munitions  frag¬ 
ments)  clutter  have  rendered  conventional  UXO  cleanup  efforts  both  expensive 
and  ineffective.  A  longer-term  objective  of  this  demonstration  is  to  collect  and 
archive  high-quality,  georeferenced  data  to  support  future  sensor  development 
and  improvements  in  UXO  analysis  technologies. 

The  performance  objectives  for  this  demonstration  are  as  follows: 

a.  To  conduct  surveys  of  one  1 -hectare  area  and  ten  30-m  by  30-m  areas 
within  Kaho’olawe  Island  under  very  difficult  but  realistic  target/ 
geologic  clutter/  man-made  clutter  scenarios  while  operating  as 
efficiently  as  possible  (minimizing  time,  manpower,  and  costs). 

b.  To  analyze  survey  data  in  a  timely  manner  (prior  to  departing  Hawaii) 
and  provide  “dig  lists”  that  include  detection,  discrimination,  and 
identification  estimates  with  associated  confidence  levels  as  well  as 
georeferenced  anomaly  maps.  Note  that  unlike  the  first  phase  demon¬ 
strations  at  JPG,  this  demonstration  phase  did  not  require  that  the 
processing  be  performed  onsite. 

c.  To  detect  and  locate  all  buried  ordnance  while  minimizing  false  alarms 
resulting  from  geologic  and  man-made  clutter  sources.  Ordnance 
location  accuracy  was  originally  specified  to  be  within  a  0.5-m  halo 
(horizontally)  around  the  emplaced  munition  (this  corresponds  to 
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excavating  a  0.5-m  horizontal  radius  hole  centered  at  the  declared  loca¬ 
tion  and  striking  any  part  of  the  UXO).  Because  of  the  highly  cluttered 
environment,  difficult  site  conditions  (wind,  terrain,  heat),  and  uncer¬ 
tainties  regarding  the  actual  position  of  the  preexisting  buried  UXO 
targets,  the  performance  of  demonstrators  was  also  evaluated  using  a 
1.0-m  radius  (Appendix  C).  Refer  to  Appendix  D  for  analysis  of  mean 
location  error  scatter  plots  for  setting  the  halo  at  0.5  m. 

d.  To  provide  high  quality,  georeferenced  data  for  postdemonstration  analy¬ 
sis  and  development  of  ROC  cxuves  and  for  broader  use  in  the  UXO 
technology  development  community. 

e.  To  prepare  technical  reports  to  evaluate  and  document  performance  and 
to  aid  the  Government  in  selecting  effective  and  efficient  systems  for 
UXO  detection  and  discrimination  in  difficult  magnetic  clutter  sites  such 
as  Kaho’olawe. 


Physical  Setup  and  Operation 

Because  Kaho’olawe  Island  contains  numerous  live  UXO,  rough  terrain,  and 
some  areas  of  thick  vegetation,  the  sites  available  for  testing  and  demonstration 
were  very  limited.  As  a  result,  this  project  was  restricted  to  operating  in  previ¬ 
ously  cleared  sites  that  fell  within  the  existing  QC  and  QA  areas.  Operating  under 
these  constraints,  personnel  from  NAVEODTECHDIV  and  the  U.S.  Army 
Engineer  Research  and  Development  Center  (ERDC),  with  support  from  onsite 
Navy  and  Contractor  personnel,  set  up  a  Calibration  site  at  the  QC  area  and  a 
limited  set  of  demonstration  scenarios  at  the  QA  area  as  shown  in  Chapter  3. 

The  UXO  targets  for  the  calibration  and  demonstration  areas  consisted  of  the 
following: 

a.  Inert  UXO  items  that  have  been  previously  used  at  Kaho’olawe  for 
QA/QC  purposes  and  whose  age,  weathering,  and  condition  can  vary. 

b.  Pristine  inert  munitions  from  the  Standard  UXO  Test  Site  Program  which 
have  been  fabricated/procured  to  be  as  identical  as  feasible. 

Several  demonstration  grids  were  prepared  so  that  the  only  UXO  targets  were 
selected  from  this  set  of  standard  test  targets.  Representative  samples  of  ordnance 
emplaced  at  the  calibration  site  were  made  available  to  demonstrators  for  view¬ 
ing  and  for  conducting  free-air  and  buried  measurements  during  the  demonstra¬ 
tion  phase. 

The  QC  grids  were  expanded  to  allow  installation  of  the  new  UXO  items 
obtained  from  the  standardized  UXO  Test  Site  stockpile.  The  demonstration 
areas  were  prepared  by  trimming  and  removing  unwanted  trees  and  tall  grasses 
and  by  watering  around  any  newly  excavated  areas  to  reduce  evidence  of  site 
disturbance.  No  other  physical  alterations  were  made  to  the  existing  ranges  as 
part  of  the  site  preparation  activities  conducted  during  3  through  26  September 
2001. 
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Descriptions  of  the  inert  UXO  targets  and  the  clutter  items  used  for  this 
demonstration  are  included  in  the  Site  Preparation  Plan  (NAVEODTECHDIV 
2001b)  which  is  available  online  as  described  in  Appendix  B.  Photographs, 
descriptions,  dimensions,  and  emplacement  information  of  each  target  and  clutter 
item  are  available  as  part  of  the  information  archived  at  this  site.  Briefly,  the 
UXO  targets  ranged  from  20-mm  projectiles  buried  near  the  surface,  to  226.8  kg 
(500-lb)  practice  bombs  buried  to  2.5  m  below  the  surface.  Clutter  items 
emplaced  ranged  from  small  (less  than  0.5  kg)  to  large  (over  5  kg)  munitions 
fragments  retrieved  during  UXO  cleanup  operations  on  Kaho’olawe. 

Based  on  concerns  raised  during  previous  UXO  detection  and  discrimination 
demonstrations,  no  unrealistic  (fabricated)  clutter  items  were  used;  instead  actual 
munitions  fragments  from  past  Kaho’olawe  cleanup  operations  were  used.  Also, 
based  on  previous  JPG  experience,  all  inert  UXO  targets  that  had  not  been  previ¬ 
ously  fired  or  air  dropped  were  demagnetized  prior  to  emplacement  to  simulate 
the  magnetic  properties  of  ordnance  that  has  been  employed.  In  addition,  precau¬ 
tions  were  taken  during  target  emplacement  to  minimize  surface  disturbances 
(e.g.,  “bathtub”  effect)  that  could  alert  the  demonstrators  to  the  presence  of  a 
buried  object. 


Sampling  Procedures 

The  Demonstration  Work  Plan  describes  the  procedures  required  for  each  of 
the  demonstrations.  Demonstrators  were  responsible  for  developing  their  specific 
survey  plans  (including  lane  spacing,  sampling  rate,  number  of  channels 
recorded,  calibration  methods,  etc.)  and  these  procedures,  together  with  their 
analysis  techniques,  are  described  in  Appendix  E. 

Each  of  the  demonstrators  was  allotted  one  3 -week  period  (Monday  through 
Thursday)  during  17  September  through  30  November  2001  to  complete  the 
following  activities: 

a.  System  calibration  and  signature  data  collection  activities  at  the 
calibration  area  during  the  first  week. 

b.  Algorithm  development  and  testing  on  Maui  during  the  second  week. 

c.  Field  surveys  of  the  demonstration  areas  and  data  analysis/preparation  of 
dig  lists  and  georeferenced  maps  during  the  third  week. 


Analytical  Procedures 

The  evaluation  factors,  metrics,  products,  and  procedures  related  to  this 
demonstration  are  described  in  the  Demonstration  Workplan 
(NAVEODTECHDIV  2001a)  and  include  the  following  information: 

Factors  that  were  measured  and  evaluated  as  part  of  this  demonstration  include: 

a.  Equipment  setup  and  calibration  time  and  man-hour  requirements. 
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b.  Actual  survey  time  and  man-hour  requirements  for  each  of  the 
demonstration  test  areas. 

c.  Downtime  resulting  from  system  malfunctions  and  maintenance 
requirements. 

d.  Reacquisition/resurvey  time  and  man-hour  requirements  (if  any). 

e.  Prioritized  dig  lists  with  associated  signal  strength  and  confidence  levels. 

f.  Discrimination  capability  (ability  to  separate  detected  anomalies  into 
UXO  and  non-UXO  objects). 

g.  Identification  capability  (ability  to  classify  UXO  targets  by  class  (e.g., 
mortar,  projectile)  and  type  (e.g.,  105-mm  projectile). 

h.  Predicted  target  location  accuracy  (including  depth  estimates). 

i.  Georeferenced  anomaly  maps. 

j.  Probability  of  Detection:  P(det). 

k.  Probability  of  Discrimination:  P(disc). 

/.  Probability  of  False  Positives  (Pfp)  resulting  only  from  the  emplaced 
clutter  items. 

m.  False  Alarm  Rates  (FAR)  resulting  from  nonemplaced  items  (e.g., 
geology  and/or  unknown  items). 

«.  Total  False  Alarm  Rates  (Total  FAR)  resulting  from  emplaced  clutter 
items  and  nonemplaced  clutter  items. 

0.  Operational  costs. 

The  method  for  determining  and  documenting  items  a,  b,  and  c  consisted  of 
the  Govermnent  onsite  representatives  tracking  and  recording  the  number  of 
personnel  and  time  spent  performing  each  of  these  tasks.  Adequate  rest  and 
lunch/snack  breaks  were  provided  and  these  times  were  included  in  the  per¬ 
formance  metrics  calculations.  If  the  demonstrator  determined  that  he  needed  to 
resurvey  any  part  of  the  test  areas  or  any  previously  detected  anomalies,  all  setup, 
calibration,  survey,  dovratime,  and  reacquisition  times  and  man-hour  require¬ 
ments  were  recorded  individually  (as  in  items  a  through  c)  but  were  compiled 
separately  as  reacquisition/resurvey  time  (item  d). 

Development  and  evaluation  of  items  e  through  /  were  as  follows: 

(1)  Each  demonstrator  was  to  combine  the  EM  sensor  data  with  the 
GPS  and  other  position  information  to  develop  2-D  anomaly  maps 
(item  i)  of  a  test  area  consisting  of  four  contiguous  30-m  by  30-m 
grids  (Area  A),  a  1 -hectare  contiguous  test  area  (Area  B),  and  a  test 
area  consisting  of  six  30-m  by  30-m  grids  (Area  C)  as  shown  in 
Figure  12.  These  anomaly  maps,  together  with  the  corresponding 
digital  geophysical  sensor  data  were  to  be  analyzed  to  identify  all 
detected  anomalies  that  could  potentially  be  a  buried  UXO  target  for 
each  of  the  test  areas.  All  of  these  anomalies  were  to  be  tabulated 
into  one  preliminary  dig  sheet  for  each  test  area  and  were  to  include 
a  suitable  “signal  strength”  value  determined  and  defined  by  each  of 
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the  demonstrators  (e.g.,  1,100  ppm  @  930  Hz/quadrature  phase  for 
the  GEM-3  system).  Each  demonstrator  was  required  to  submit  a 
detailed  description  of  and  rationale  for  selecting  this  parameter  as 
part  of  the  prioritized  dig  list  submissions  described  later  in  this 
document.  The  objective  of  the  detection  stage  was  to  include  as 
many  anomalies  in  these  lists  as  required  to  ensure  as  high  a  P(det) 
as  possible  for  the  full  range  of  UXO  targets  considered. 

(2)  Each  anomaly  in  each  list  was  then  to  be  further  analyzed  to  develop 
the  final  dig  sheets  as  illustrated  in  Table  1.  The  demonstrators  were 
to  refine  the  location  (x,  y)  and  estimate  the  depth  (z)  of  each  object, 
attempt  to  separate  (discriminate)  UXO  from  clutter  items,  identify 
UXO  by  class  and  type  (if  possible),  and  rank  the  list  in  the 
following  descending  order:  UXO  -  high  confidence,  UXO  - 
medium  confidence,  UXO  —  low  confidence.  Clutter  —  low 
confidence,  Clutter  -  medium  confidence,  and  Clutter  -  high 
confidence.  In  addition,  the  list  was  to  include  predicted  ordnance 
class  and  type  (e.g.,  mortar/8 1mm)  for  all  anomalies  declared  as 
UXO  with  high  and  medium  confidence  levels,  and,  if  possible, 
UXO  orientation  (Azimuth  and  Inclination). 


Table  1 

Sample  Dig  List 

IdIGLIST:  1  Demonstrator:  EMMS  Test  Area:  A  Including  20/40  mm  ?:  NO 


Ranking 


001 


Northing 

m 


4309738.557 


Easting 

m 


641594.2038 


Depth 

m 


0.9144 


Type 

ordnance 

/clutter 


ordnance 


Confidence 


high 


Size/ 

Weight 


large 


Azimuth 

deg 


180 


Inclination 

deg 


20 


Class 


projectile 


Type 


152  mm 


050 


4309689.964 


641519.4151 


0.89042 


ordnance 


low 


small 


projectile 


unknown 


165 


4309700.031 


641516.8877 


0.82296 


clutter 


high 


medium 


frag 


(3)  Each  demonstrator  was  then  required  to  select  a  point  (threshold)  on 
each  prioritized  list  where  he  would  recommend  that  all  objects  at  or 
above  that  point  be  excavated  and  the  remainder  left  in  place.  We 
refer  to  this  as  the  stop-dig-point.  The  goal  of  this  step  was  to  maxi¬ 
mize  the  number  of  UXO  targets  above  the  threshold  while  mini¬ 
mizing  the  number  of  clutter  items.  In  order  to  add  realism  to  this 
demonstration,  the  following  cost  penalties  were  to  be  applied  to 
this  product:  For  every  clutter  item  selected  for  “digging,”  a  $200 
cost  penalty  was  assigned  (the  average  cost  of  excavating  items  at 
actual  UXO  remediation  sites).  If  the  demonstrator  included  any 
UXO  items  in  the  “no  dig”  portion  of  the  list,  it  was  assumed  that 
the  area  (i.e.,  either  the  1-hectare  area,  four  30-m  grids,  or  six  30-m 
grids)  would  fail  the  Quality  Assurance  and/or  regulatory 
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acceptance,  and  a  cost  penalty  equal  to  the  cost  of  a  resurvey  was 
assigned.  Missed  UXO  targets,  that  is  those  not  included  in  the  dig 
lists  either  as  clutter  or  UXO,  were  also  penalized  the  cost  of  a 
resurvey.  Note  that  the  same  resurvey  penalty  was  applied  only  once 
for  each  area  whether  only  one  or  any  number  of  UXO  items  were 
left  in  the  ground  through  misdiscrimination,  or  by  failing  to  detect 
it,  or  a  combination  of  both. 

(4)  In  order  to  better  approximate  real-world  UXO  cleanup  operations, 
the  Government  offered  to  furnish  ground-truth  information  of  Area 
A  within  24  hr  of  a  demonstrator’s  submitting  prioritized  dig  lists 
for  that  specified  portion  of  the  test  area.  This  procedure  was 
intended  to  correspond  to  the  additional  information  that  is  normally 
available  to  the  UXO  survey  contractors  when  surveyed  grids  are 
excavated.  It  should  be  noted  that  groundtruth  corresponding  only  to 
anomalies  included  in  the  prioritized  dig  lists  were  provided.  No 
information  on  missed  targets  was  made  available  until  the  full 
groundtruth  was  released  after  the  completion  of  demonstrations.  In 
order  to  provide  this  information  as  early  as  possible  during  the 
blind  demonstrations,  the  following  procedure  was  followed.  At  the 
beginning  of  the  demonstration  phase,  each  demonstrator  was 
directed  to  survey  Area  A,  consisting  of  four  30-m  by  30-m  grids. 
The  demonstrator  was  then  to  proceed  to  develop  a  prioritized  dig 
list  of  this  area  while  his  survey  crew  continued  to  collect  field 
survey  data  of  the  remaining  test  areas.  As  soon  as  the  demonstrator 
submitted  the  prioritized  dig  list,  the  Government  representative 
evaluated  the  results  and  provided  groundtruth  information  on  the 
declared  target  and  clutter  items.  The  demonstrator  could  then  use 
that  information  to  modify  his  analysis  and/or  survey  techniques 
during  the  remainder  of  the  blind  demonstration.  The  goal  was  for 
each  demonstrator  to  have  this  information  prior  to  starting  the 
analysis  of  the  1 -hectare  site.  It  should  be  noted  that  the  majority  of 
demonstrators  did  not  submit  the  dig  lists  for  Area  A  until  the  end  of 
the  demonstrations  and,  therefore,  did  not  receive  groundtruth 
information  to  aid  in  the  analysis  of  Areas  B  and  C. 

(5)  Items  j  through  n  were  calculated  from  the  prioritized  dig  lists  as 
follows:  Maximum  achievable  P(det)  for  each  area  was  ealeulated 
as  the  number  of  items  in  the  entire  list  that  correspond  to  emplaced 
UXO  targets  (even  though  they  may  have  been  misdiseriminated  as 
clutter)  divided  by  the  actual  number  of  UXO  targets  emplaced  in 
that  site.  Similarly,  maximum  achievable  P(disc)  was  calculated  as 
the  number  of  anomalies  in  the  dig  list  that  were  correctly  classified 
as  UXO  divided  by  the  total  number  of  emplaced  UXO  targets.  In 
addition,  the  single  point  probability  of  discrimination  (P(disc))  was 
determined  by  calculating  the  number  of  actual  UXO  targets  that  are 
correctly  classified  as  UXO  and  are  included  in  the  list  at  or  above 
the  specified  dig  point.  The  operating  (single  point)  FAR  was 
calculated  as  the  number  of  items  per  surveyed  area  that  were 
included  above  the  dig  threshold  and  which  did  not  correspond  to 
emplaced  target  or  emplaced  clutter  locations.  FAR  is  therefore  a 
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measure  of  the  false  positives  because  of  natural  geologic/ 
environmental  factors  and  any  preexisting  metal  objects.  In 
addition,  Pfp  was  calculated  as  the  ratio  of  the  number  of  clutter 
items  declared  as  UXO  to  the  number  of  clutter  items  emplaced. 
Total  False  Alarm  Rates  (item  n)  were  computed  by  combining  both 
false  alarm  sources  that  make  up  items  /  and  m.  ROC-like  curves 
were  developed  by  the  Government  by  varying  the  dig  threshold 
until  the  maximum  P(det)  and  P(disc)  were  reached,  and  plotting 
P(det),  P(disc)  as  the  ordinate  and  Pfp  and  FAR  as  the  abscissa  at 
each  increment.  ROC-like  curves  of  P(det)  and  P(disc)  vs.  Total 
FAR  were  also  developed  by  using  the  specified  “signal  strength” 
parameter  as  the  thresholding  variable.  Performance  comparisons 
between  systems  include  using  the  ROC-like  curves  to  determine 
Pfp  and  FAR  at  the  P(det)  required  for  Kaho’olawe  Tier  II  clearance 
(85  percent). 

(6)  After  each  demonstrator  had  submitted  the  dig  sheets  described 
above,  a  total  of  three  dig  lists,  he/she  was  given  the  opportunity  to 
reanalyze  the  data  from  the  1 -hectare  site  and  the  six  30-m  by  30-m 
grids  to  develop  two  additional  prioritized  dig  sheets  that  take  into 
account  only  targets  larger  than  40-mm  projectiles.  All  dig  sheets 
were  to  be  submitted  to  the  Government  representative  within  3 
days  after  completing  the  field  demonstration  and  prior  to  departing 
Hawaii.  After  all  field  demonstrations  were  completed,  the  ESTCP 
Program  Office  provided  each  of  the  demonstrators  with  the 
complete  groundtruth  for  all  of  the  test  areas  at  both  JPG  and 
Kaho’olawe.  Each  demonstrator  was  then  required  to  reanalyze 
his/her  results,  identify  problems  and  potential  improvements,  and 
submit  a  self-evaluation  draft  report  to  the  ESTCP  Program  Office. 
This  draft  report  will  be  due  no  later  than  2  months  after  release  of 
the  groundtruth.  The  ESTCP  Program  Office  will  review  the  draft 
report  and  return  it  to  the  demonstrator  with  comments.  A  final 
report  will  be  due  no  later  than  30  days  after  receiving  comments 
from  the  ESTCP  Program  Office. 

(7)  Item  o.  Operational  costs  were  estimated  using  the  cost  factors  and 
procedures  described  in  Chapter  6. 
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5  Performance  Assessment 


Introduction 

In  accordance  with  the  Demonstration  Plan,  eaeh  of  the  demonstrators  was 
responsible  for  determining  the  best  method  of  employing  his  system  in  order  to: 
(a)  ensure  full  coverage  of  each  demonstration  area,  (b)  collect  high-quality 
sensor  data  to  support  deteetion  and  discrimination  requirements,  (c)  achieve 
high  production  rates,  and  (d)  minimize  man-hour  requirements  and  costs.  All 
demonstrators  were  able  to  complete  the  field  surveys  within  the  allotted  time 
periods. 

There  was  a  wide  range  in  the  demonstrators’  survey  data  quality,  data 
density,  quality  of  analysis,  and  complianee  with  the  data  submission  require¬ 
ments  speeified  in  the  Demonstration  Plan  (NAVEODTECHDIV  2001a).  For 
example,  a  number  of  demonstrators  failed  to  include  required  dig  list  informa¬ 
tion  such  as  recommended  stop  dig  point,  appropriate  confidence  levels,  and 
signal  strength  levels,  and  most  demonstrators  failed  to  reanalyze  their  data  and 
prepare  dig  lists  that  excluded  the  small  20  mm  and  40  mm  targets.  This  lack  of 
adherence  to  the  requirements  of  the  Demonstration  Plan  has  made  the  interpre¬ 
tation  of  results  and  adequate  across-demonstrator  performance  comparisons  very 
difficult.  This  Chapter  presents  a  summary  of  the  data  submitted  by  the  demon¬ 
strators  and  the  Government’s  assessment  of  their  performance. 

It  should  be  noted  that  because  of  the  very  high  numbers  of  false  alarms 
submitted  by  all  demonstrators,  the  Government  was  not  able  to  fully  investigate 
the  sources  of  all  of  them.  Nevertheless,  during  April  2002,  NAVEODTECHDIV 
personnel  conducted  extensive  surveys  and  excavation  activities  in  the  ealibra- 
tion  and  demonstration  areas  in  order  to  verify  the  emplaced  target  loeations  and 
to  attempt  to  identify  the  sources  of  a  large  percentage  of  the  false  alarms. 
Information  from  these  postdemonstration  activities  was  incorporated  into  the 
groundtruth  data  used  to  evaluate  the  demonstrators’  performance. 

During  the  verification  of  the  emplaced  target  locations,  the  locations  of  the 
following  targets  were  updated:  lA  87,  lA  204,  lA  225,  IE  95,  IE  103,  2E  96, 
3D  691, 4D  116,  4D  169,  and  4D  89.  All  of  these  targets  were  previously 
emplaced  targets  on  the  site. 

While  identifying  sources  of  false  alarms,  16  items  were  added  to  the 
groundtruth.  Eleven  of  the  items  were  considered  to  be  no-grade  items  and  were 
removed  from  evaluation.  Their  labels  are  1C  C8,  1C  C48,  1C  C49,  ID  A7,  ID 
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A1 1,  2A  C15,  2B  C34,  2B  C42,  3B  B16,  3E  B6,  and  3E  B7.  The  other  five  items 
were  graded  as  ordnance.  Their  labels  are  3B  B14,  3B  B17,  4B  B3,  5B  B32,  and 
5C  B36. 


Anomaly  Maps 

Figures  16  through  31  show  the  geo-referenced  anomaly  maps  produced  by 
each  of  the  systems  used  during  these  demonstrations.  Parsons  did  not  submit  an 
anomaly  map  of  Area  A  for  EM-61  Digital;  however,  they  did  submit  a  dig  sheet 
for  this  area. 


Detection  Results 

Table  2  summarizes  the  number  of  UXO  targets  detected  within  0.5-m  circu¬ 
lar  error  by  each  demonstrator.  The  maximum  achievable  probability  of  detec¬ 
tion,  P(det),  is  also  given  in  the  table.  The  P(det)  is  calculated  as  the  number  of 
declared  items  that  correspond  to  emplaced  UXO  targets  (even  though  they  may 
have  been  misclassified  as  clutter)  divided  by  the  actual  number  of  UXO  targets 
emplaced  in  the  demonstration  site. 


Table  2 

P(det)  bv  Area  within  0.5  m 

Within  0.5  m  I 

Area  A 

Area  B 

Area  C 

Total  I 

Number  of  Actual 
Targets 

24 

81 

34 

139 

NAEVA 

Targets  Detected 

13 

33 

10 

56 

P(det) 

0.542 

0.407 

0.294 

0.403 

GTL 

Targets  Detected 

10 

30 

3 

43 

Pfdet) 

0.417 

0.370 

0.088 

0.309 

Geophex 

Targets  Detected 

16 

33 

10 

59 

P(det) 

0.667 

0.407 

0.294 

0.424 

NRL 

Targets  Detected 

8 

17 

6 

31 

P(det) 

0.333 

0.210 

0.176 

0.223 

NRL  without  20/40  mm 

Targets  Detected 

9 

20 

8 

37 

P(det) 

0.375 

0.247 

0.235 

0.266 

Parsons  EM-61 

EM  and  Flag 

Targets  Detected 

12 

33 

15 

60 

Pfdet) 

0.500 

0.407 

0.441 

0.432 

Parsons  EM-61  Digital 

Targets  Detected 

12 

25 

15 

52 

Pfdetl 

0.500 

0.309 

0.441 

0.374 

Parsons  TM-5  EMU 

Targets  Detected 

6 

13 

1 

20 

P(det) 

0.250 

0.160 

0.029 

0.144 
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Figure  16.  NAEVA  EM-63  survey  of  Area  A 
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Figure  17.  NAEVA  EM-63  survey  of  Area  B 
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Figure  18.  NAEVA  EM-63  survey  of  Area  C 
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Figure  19.  GTL  TM-5  EMU  survey  of  Area  A 
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Figure  20.  GTL  TM-5  EMU  survey  of  Area  B 
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Figure  21 .  GTL  TM-5  EMU  survey  of  Area  C 


Figure  22.  Geophex,  Ltd.  GEM-3  survey  of  Areas  A,  B,  and  C 
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Figure  23.  NRL  EMMS  survey  of  Areas  A,  B,  and  C 

BEST  AVAILABLE  COPY 
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Figure  24.  Parsons  EM  and  Flag  survey  of  Area  A 
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Figure  26.  Parsons  EM  and  Flag  survey  of  Area  C 
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Figure  28.  Parsons  EM-61  Digital  survey  of  Area  C 
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Figure  30.  Parsons  TM-5  EMU  survey  of  Area  B 
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Table  3  summarizes  the  detection  results  achieved  when  the  small  UXO 
targets  (20-mm  and  40-mm  projectiles)  were  excluded  from  the  evaluation.  The 
detection  results  within  a  1.0-m  circular  error  can  be  found  in  Appendix  C.  These 
results  indicate  that  no  demonstrator  was  able  to  achieve  the  Kaho’olawe  Tier  11 
clearance  requirements  of  0.85  P(det)  with  a  0.5-m  location  accuracy  at  any  of 
the  three  demonstration  areas.  Only  when  the  requirements  were  relaxed  by 
expanding  the  allowable  position  error  to  1.0  m  and  also  deleting  the  smaller 
UXO  targets  did  any  of  the  demonstrators  meet  the  P(det)  requirements  (see 
Appendix  C,  Table  C1-C2).  Even  then,  acceptable  P(det)  levels  were  only 
obtained  in  Area  A  which  had  considerably  lower  levels  of  geologic  noise  and 
metallic  clutter  than  the  other  two  Areas. 


Table  3 

P(det)  by  Area  within  0.5  m  and  without  20/40  mm 


Within  0.5  m 

Totai 

Area  A 

Area  B 

Area  C 

Number  of  Actual 
Targets  without 

20/40  mm 

19 

55 

28 

102 

NAEVA 

Targets  Detected 

11 

26 

9 

46 

P(det) 

0.579 

0.473 

0.321 

0.451 

GTL 

Targets  Detected 

9 

24 

3 

36 

P(det) 

0.474 

0.436 

0.107 

0.353 

Geophex 

Targets  Detected 

13 

28 

9 

50 

P(det) 

0.684 

0.509 

0.321 

0.490 

NRL 

Targets  Detected 

7 

17 

6 

30 

P(det) 

0.368 

0.309 

0.214 

0.294 

NRL  without  20/40  mm 

Targets  Detected 

8 

20 

8 

36 

P(det) 

0.421 

0.364 

0.286 

0.353 

Parsons  EM-61 

EM  and  Flag 

Targets  Detected 

11 

26 

15 

52 

P(det) 

0.579 

0.473 

0.536 

0.510 

Parsons  EM-61  Digital 

Targets  Detected 

10 

24 

15 

49 

P(det) 

0.526 

0.436 

0.536 

0.480 

Parsons  TM-5  EMU 

Targets  Detected 

6 

12 

1 

19 

P(det) 

0.316 

0.218 

0.036 

0.186 

Table  4  shows  the  number  of  UXO  targets  detected  within  0.5  m  by  each 
demonstrator  in  each  30-  x  30-m  grid.  Shaded  entries  indicate  the  highest  number 
of  detections  for  that  grid.  The  last  column  of  the  table  indicates  the  number  of 
UXO  targets  not  detected  by  any  of  the  demonstrators.  Tables  5  through  12  show 
detections  by  ordnance  t5q)e  for  each  demonstrator.  Sixteen  different  ordnanee 
types  are  listed  in  each  table.  No  UXO  targets  of  the  other  two  types  listed  in  the 
demonstration  plan  (BDU  3  practice  bomb  and  3 -in.  projectile)  were  emplaced 
for  this  demonstration. 


Chapter  5  Performance  Assessment 


45 


Target  Location  Performance 

In  order  to  evaluate  the  demonstrator’s  field  survey  data,  a  computer  program 
was  compiled  in  SAS  software.  This  program  calculated  an  evaluation  variable 
by  sequentially  going  from  the  top  of  the  prioritized  dig  list  and  determining  if 
each  object  on  the  list  (whether  classified  as  target  or  clutter)  corresponds  to  an 
emplaced  target  location  within  0.5  m  (a  detection)  or  not  (a  false  alarm). 
Similarly,  Appendix  C  considers  a  detection  if  it  is  within  1.0  m. 

The  location  (x,y)  performance  (Table  13)  of  each  of  the  seven  demonstra¬ 
tors  was  evaluated  by  comparing  each  item  in  the  dig  list  with  the  groundtruth, 
determining  the  closest  item  to  an  emplaced  UXO  target  location,  and  computing 
the  error.  The  ability  to  locate  clutter  items  was  not  one  of  the  evaluation  criteria, 
but  the  raw  data  were  available  for  such  analysis  if  deemed  useful. 
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Table  5 

UXO  Found  -  NAEVA 


□escriDtion 


2.25-in. 


2.75-in. 


5-in. 


20-mm 


40-mm 


60-mm 


81-mm 


105-mm 


BDU-33 


Mk-3 


Mk-81 


Mk-82 


Mk-83 


Mk-106 


LAAW 


SMAW 


Total 


Actual  Tarqets 


B  C 


Within  0.5  m 


c 

Total 

0 

2 

1 

6 

0 

2 

0 

1 

1 

9 

1 

8 

1 

7 

1 

3 

2 

6 

2 

6 

0 

0 

1 

3 

0 

1 

0 

2 

0 

1 

0 

0 

10 

56 

Table  6 

UXO  Found  -  GTL 


Descr  Dt  on 


2.25-in. 


2.75-in. 


5-in. 


20-mm 


40-mm 


60-mm 


81-mm 


105-mm 


BDU-33 


Mk-3 


Mk-81 


Mk-82 


Mk-83 


Mk-106 


LAAW 


SMAW 


Total 


Actual  Tarqets 


Within  0.5  m 


0 

2 

0 

2 

2 

0 

1 

3 

0 

3 

0 

3 

1 

1 

0 

2 

0 

6 

0 

6 

2 

4 

1 

7 

1 

5 

1 

7 

1 

1 

0 

2 

2 

1 

0 

3 

0 

2 

0 

2 

0 

0 

0 

0 

0 

2 

0 

2 

0 

1 

0  , 

1 

0 

3 

0 

3 

1 

0 

0 

1 

0 

0 

0 

0 

10 

30 

3 
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Table  7 

UXO  Found  -  Geophex  &  Geophex  without  20/40  mm 


Description 

Area 

Actuai  Taraets 

Within  0.5  m 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

2 

0 

2 

2.75-in. 

4 

5 

4 

13 

2 

1 

1 

4 

5-in. 

2 

4 

2 

8 

1 

3 

0 

4 

20-mm 

4 

12 

1 

17 

2 

1 

0 

3 

40-mm 

1 

14 

5 

20 

1 

4 

1 

6 

60-mm 

2 

11 

3 

16 

1 

3 

2 

6 

81-mm 

1 

8 

3 

12 

1 

6 

1 

8 

105-mm 

1 

3 

1 

5 

1 

3 

0 

4 

BDU-33 

3 

4 

3 

10 

3 

2 

2 

7 

Mk-3 

1 

4 

3 

8 

1 

3 

2 

6 

Mk-81 

1 

2 

1 

4 

1 

0 

0 

1 

Mk-82 

1 

4 

7 

12 

0 

2 

1 

3 

Mk-83 

0 

1 

0 

1 

0 

1 

0 

1 

Mk-106 

0 

4 

0 

4 

0 

3 

0 

3 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

1 

0 

0 

1 

Total 

24 

81 

34 

139 

16 

33 

10 

59 

Note:  Geophex  without  20/40  mm  detected  the  same  ordnance  items. 


Table  8 

UXO  Found  -  NRL 

Description 

Area  II 

Actual  Targets 

Within  0.5  m  II 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

2 

0 

2 

2.75-in. 

4 

5 

4 

13 

1 

1 

1 

3 

5-in. 

2 

4 

2 

8 

1 

2 

0 

3 

20-mm 

4 

12 

1 

17 

1 

0 

0 

1 

40-mm 

1 

14 

5 

20 

0 

0 

0 

0 

60-mm 

2 

11 

3 

16 

1 

2 

1 

4 

81-mm 

1 

8 

3 

12 

0 

3 

0 

3 

105-mm 

1 

3 

1 

5 

0 

2 

0 

2 

BDU-33 

3 

4 

3 

10 

3 

1 

1 

5 

Mk-3 

1 

4 

3 

8 

0 

1 

0 

1 

Mk-81 

1 

2 

1 

4 

0 

0 

1 

1 

Mk-82 

1 

4 

7 

12 

0 

1 

2 

3 

Mk-83 

0 

1 

0 

1 

0 

1 

0 

1 

Mk-106 

0 

4 

0 

4 

0 

2 

0 

2 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

0 

0 

0 

0 

Totai 

24 

81 

34 

139 

8 

17 

6 

31 
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Table  9 

UXO  Found  -  NRL  without  20/40  mm 


DescriDtion 


2.25-in. 


2.75-in. 


5-in. 


20-mm 


40-mm 


60-mm 


81-mm 


105-mm 


BDU-33 


Mk-3 


Mk-81 


Mk-82 


Mk-83 


Mk-106 


LAAW 


SMAW 


Totai 


Actual  Targets 


Within  0.5  m 


A 

B 

C 

Totai 

0 

2 

0 

2 

1 

1 

1 

3 

1 

2 

0 

3 

1 

0 

0 

1 

0 

0 

0 

0 

2 

3 

1 

6 

0 

3 

1 

4 

0 

2 

0 

2 

3 

1 

1 

5 

0 

2 

0 

2 

0 

0 

1 

1 

0 

1 

2 

3 

0 

1 

0 

1 

0 

2 

0 

2 

1 

0 

0 

1 

0 

0 

1 

1 

g 

19 

8 

36 

Table  10 

UXO  Found  -  Parsons  EM-61  EM  &  Fla 


Area 


Actual  Targets 


C 


Descr  Dtlon 


2.25-in. 


2.75-in. 


5-in. 


20-mm 


40-mm 


60-mm 


81-mm 


105-mm 


BDU-33 


Mk-3 


Mk-81 


Mk-82 


Mk-83 


Mk-106 


LAAW 


SMAW 


Totai 


Within  0.5  m 


B  C 
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Table  13 

UXO  Target  Location  (x,  y)  Estimation  Performance  of  the 
Demonstrators 

Area 

Demonstrator 

Minimum 
Error,  m 

Maximum 
Error,  m 

Mean  Error, 
m 

Standard 

Deviation 

A 

NAEVA 

0.065 

0.968 

0.444 

0.233 

GTL 

0.139 

0.856 

0.395 

0.249 

Geophex 

0.105 

0.941 

0.357 

0.217 

NRL 

0.146 

0.842 

0.395 

0.213 

NRL  without  20/40  mm 

0.123 

0.911 

0.406 

0.271 

Parson  (EM-61)  EM  and  Flag 

0.074 

0.737 

0.403 

0.198 

Parson  EM-61  Digital 

0.020 

0.993 

0.513 

0.270 

Parson  TM-5  EMU 

0.108 

0.534 

0.322 

0.161 

B 

NAEVA 

0.071 

0.960 

0.444 

0.232 

GTL 

0.036 

0.877 

0.400 

0.258 

Geophex 

0.025 

0.984 

0.403 

0.246 

NRL 

0.080 

0.979 

0.445 

0.279 

NRL  without  20/40  mm 

0.080 

0.999 

0.530 

0.289 

Parson  (EM-61)  EM  and  Flag 

0.040 

0.991 

0.394 

0.247 

Parson  EM-61  Digital 

0.028 

0.970 

0.435 

0.259 

Parson  TM-5  EMU 

0.200 

0.886 

0.629 

0.302 

C 

NAEVA 

0.074 

0.992 

0.505 

0.302 

GTL 

0.260 

0.981 

0.673 

0.203 

Geophex 

0.114 

0.820 

0.394 

0.244 

NRL 

0.234 

0.718 

0.434 

0.199 

NRL  without  20/40  mm 

0.234 

0.771 

0.461 

0.193 

Parson  (EM-61)  EM  and  Flag 

0.064 

0.876 

0.405 

0.225 

Parson  EM-61  Digital 

0.074 

0.863 

0.419 

0.225 

Parson  TM-5  EMU 

0.089 

0.914 

0.438 

0.273 

The  ability  of  the  demonstrators  to  estimate  the  depth  of  the  UXO  targets  is 
summarized  in  Table  14.  These  results  indicate  that,  while  the  performance  of 
each  demonstrator  varied  significantly  between  each  demonstration  area,  the 
mean  depth  estimation  errors  were  within  the  0.5-m  allowable  error.  Overall,  the 
PARSON  TM-5  EMU  system  achieved  the  best  depth  estimation  accuracy  for 
Areas  A  and  C.  The  accuracy  was  the  worst  in  Area  B,  with  the  largest  maximum 
and  mean  depth  estimation  errors  demonstrated.  The  ranking  of  the  demonstra¬ 
tors  differed  in  each  area. 


Discrimination  Results 

Tables  15  through  30  illustrate  the  comparison  of  the  demonstrator’s  decla¬ 
rations  versus  the  groundtruth  emplaced  targets.  Table  15  summarizes  the  per¬ 
formance  of  the  demonstrators  over  the  entire  demonstration  site  within  the  QA 
Range.  In  this  table,  the  ordnance  items  left  in  the  ground  represent  the  items 
declared  by  the  demonstrator  as  clutter  with  high  confidence  that  are  actually 
groundtruth  emplaced  ordnance  plus  the  number  of  undetected  ordnance  items. 
The  variable  Correct  Discrimination  is  the  total  of  the  correctly  identified  ord¬ 
nance  items.  The  False  Alarm  number  is  the  total  of  the  other  detections  plus  the 
total  of  the  Groundtruth  Fragment  Matches  minus  the  number  of  objects 
identified  as  Clutter  with  High  confidence. 
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Table  14 


UXO  Target  Depth  Estimation  Performance  of  the  Demonstrators 


Area 

Demonstrator 

Minimum 
Error,  m 

Maximum 
Error,  m 

Mean  Error, 
m 

Standard 

Deviation 

A 

Naeva 

0.010 

0.394 

0.233 

0.118 

GTL 

0.000 

0.970 

0.208 

0.198 

Geophex 

0.000 

1.250 

0.278 

0.244 

NRL 

0.015 

0.677 

0.360 

0.180 

Parson  (EM-61)  EM  and  Flag 

0.000 

1.020 

0.232 

0.237 

Parson  EM-61  Digital 

0.020 

0.550 

0.219 

0.128 

Parson  TM-5  EMU 

B 

Naeva 

0.006 

1.392 

0.218 

0.188 

GTL 

0.004 

0.560 

0.184 

0.140 

Geophex 

0.000 

1.150 

0.286 

0.255 

NRL 

0.015 

0.830 

0.370 

0.204 

Parson  (EM-61)  EM  and  Flag 

0.000 

1.832 

0.212 

0.261 

Parson  EM-61  Digital 

0.010 

0.570 

02138 

0.131 

Parson  TM-5  EMU 

C 

Naeva 

0.010 

0.552 

0.207 

0.134 

GTL 

0.010 

0.921 

0.217 

0.211 

Geophex 

0.000 

1.274 

0.223 

0.234 

NRL 

0.039 

0.652 

0.313 

0.187 

Parson  (EM-61)  EM  and  Flag 

0.000 

1.324 

0.233 

0.267 

Parson  EM-61  Digital 

0.000 

0.840 

0.202 

0.132 

Parson  TM-5  EMU 

Tables  16  through  30  show  more  detailed  breakdowns  of  the  results  by  area 
for  each  demonstrator.  The  left  most  column  of  each  of  these  tables  represents 
the  emplaced  targets.  These  items  are  matched  with  the  demonstrator’s  dig  list. 
When  the  demonstrator  reported  the  object  as  ordnance  and  the  emplaced  target 
was  a  groundtruth  ordnance  match  then  it  is  considered  a  correct  discrimination. 
The  confidence  of  the  demonstrator’s  declaration  is  used  to  determine  which 
items  from  his  dig  list  are  to  be  excavated.  In  lieu  of  a  demonstrators’  defined 
“stop  dig  point,”  only  objects  that  are  declared  as  clutter  with  high  confidence 
would  be  left  in  the  ground. 

For  example,  NAEVA  (under  0.5  m)  discrimination  performance  is  illu¬ 
strated  in  Table  16.  NAEVA  reported  295  items  as  Ordnance.  Of  that  number, 

37  actually  matched  the  ground-truth  ordnance  items  emplaced  in  the  ground, 

34  matched  the  ground-truth-emplaced  fragments,  and  224  detected  something 
other  than  the  targets  emplaced  in  the  ground.  NAEVA  reported  630  items  as 
clutter.  Only  25  actually  matched  the  Groundtruth  Fragments  emplaced  in  the 
ground,  19  matched  the  groundtruth  emplaced  ordnance  items,  and  586  detected 
something  other  than  the  targets  emplaced  in  the  ground.  NAEVA  reported  250 
(241+4+5)  with  high  confidence  of  the  630  clutter  items,  therefore,  40  percent  of 
the  identified  clutter  were  designated  as  “do  not  dig.”  There  were  five  ordnance 
items  that  would  have  been  left  in  the  ground  as  a  result  of  being  classified  as 
high  confidence  clutter.  Also,  from  Table  15,  there  were  88  undetected  ordnance 
items.  NAEVA  reported  a  total  of  810  items  which  did  not  match  any  emplaced 
ordnance,  and  59  items  that  matched  emplaced  fragments  and  56  detections  of 
emplaced  ordnances. 
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Table  15 

Summary  of  Discrimination  Performance 


Ordnance  Left  in  the 
Ground 

Correct  Discrimination 

Raise  Aiarm 
Number 

Within  0.5  m 

Within  0.5  m 

Within  0.5  m 

Num 

Percent 

Num 

Percent 

Naeva 

88 

63.31% 

51 

36.69% 

624 

GTL 

96 

69.06% 

43 

30.94% 

1,283 

Geophex 

80 

57.55% 

59 

42.45% 

772 

Geophex  without  20/40  mm 

80 

57.55% 

59 

42.45% 

772 

NRL 

108 

77.70% 

31 

22.30% 

342 

NRL  without  20/40  mm 

102 

73.38% 

37 

26.62% 

602 

Parson  (EM-61)  EM  &  Flag 

79 

56.83% 

60 

43.17% 

872 

Parson  EM-61  Digital 

87 

62.59% 

52 

37.41% 

1,405 

Parson  TM-5  EMU 

119 

85.61% 

20 

14.39% 

172 

Table  16 


Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  NAEVA 


Ground- 

truth 

Demonstrator  Classification  I 

Confidence 

QA  Range 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

381 

140 

241 

171 

31 

140 

Low 

429 

84 

345 

99 

23 

76 

Total 

810 

224 

586 

270 

54 

216 

Clutter 

High 

30 

26 

4 

9 

6 

3 

Low 

29 

8 

21 

7 

2 

5 

Total 

59 

34 

25 

16 

8 

8 

Ordnance 

High 

31 

26 

5 

8 

6 

2 

Low 

25 

11 

14 

5 

1 

4 

Totai 

56 

37 

19 

13 

7 

6 

Total 

925 

295 

630 

299 

69 

230 

Table  17 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  NAEVA 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

127 

62 

65 

83 

47 

36 

Low 

185 

32 

153 

145 

29 

116 

Total 

312 

94 

218 

228 

76 

152 

Clutter 

High 

14 

13 

1 

7 

7 

0 

Low 

13 

2 

11 

9 

4 

5 

Total 

27 

15 

12 

16 

11 

5 

Ordnance 

High 

17 

15 

2 

6 

5 

1 

Low 

16 

8 

8 

4 

2 

2 

Total 

33 

23 

10 

10 

7 

3 

Total 

372 

132 

240 

254 

94 

160 
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Table  18 

Breakdown  of  Discrimination  -  QA  Range  -  GTL 

Ground- 

truth 

Demonstrator  Classification 

Confidence 

Total 

Area  A 

Area  B 

Area  C 

Declared 

Ordnance 

Declared 

Ordnance 

Declared 

Ordnance 

Declared 

Ordnance 

Non- 

emplaced 

High 

"682 

96 

324 

262 

Low 

307 

66 

167 

74 

Medium 

234 

56 

122 

56 

Total 

1,223 

218 

613 

392 

Clutter 

High 

46 

13 

23 

10 

Low 

8 

2 

4 

2 

Medium 

6 

3 

2 

1 

Total 

60 

18 

29 

13 

Ordnance 

High 

35 

8 

25 

2 

Low 

3 

1 

1 

1 

Medium 

5 

1 

4 

0 

Total 

43 

10 

30 

3 

Total 

1,326 

246 

672 

408 

Table  19 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A-  Geophex  & 
Geophex  without  20/40  mm 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

QA  Range 

Area  A  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

104 

100 

4 

18 

18 

0 

Low 

291 

248 

43 

53 

47 

6 

Medium 

317 

281 

36 

48 

44 

4 

Total 

712 

629 

83 

119 

109 

10 

Clutter 

High 

21 

21 

0 

8 

8 

0 

Low 

13 

11 

2 

2 

2 

0 

Medium 

30 

28 

2 

5 

5 

0 

Total 

64 

60 

4 

15 

15 

0 

Ordnance 

High 

19 

19 

0 

5 

5 

0 

Low 

16 

16 

0 

4 

4 

0 

Medium 

24 

24 

0 

7 

7 

0 

Totai 

59 

59 

0 

16 

16 

0 

Total 

835 

748 

87 

150 

140 

10 
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Table  20 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Geophex  & 
Geophex  without  20/40  mm _ 


Ground- 

truth 

Demonstrator  Classification  | 

Confidence 

Area  B 

Area  C  | 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

67 

64 

3 

19 

18 

1 

Low 

174 

155 

19 

64 

46 

18 

Medium 

199 

174 

25 

70 

63 

7 

Total 

440 

393 

47 

153 

127 

26 

Clutter 

High 

10 

10 

0 

3 

3 

0 

Low 

8 

6 

2 

3 

3 

0 

Medium 

16 

14 

2 

9 

9 

0 

Total 

34 

30 

4 

15 

15 

0 

Ordnance 

High 

9 

9 

0 

5 

5 

0 

Low 

10 

10 

0 

2 

2 

0 

Medium 

14 

14 

0 

3 

3 

0 

Totai 

33 

33 

0 

10 

10 

0 

Total 

507 

456 

51 

178 

152 

26 

Table  21 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  NRL 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

QA  Range 

Area  A  | 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

10 

10 

0 

0 

0 

0 

Low 

229 

98 

131 

81 

30 

51 

Medium 

76 

60 

16 

36 

27 

9 

Total 

315 

168 

147 

117 

57 

60 

Clutter 

High 

8 

8 

0 

0 

0 

0 

Low 

11 

10 

1 

4 

4 

0 

Medium 

8 

8 

0 

2 

2 

0 

Total 

27 

26 

1 

6 

6 

0 

Ordnance 

High 

5 

5 

0 

0 

0 

0 

Low 

13 

7 

6 

2 

1 

1 

Medium 

13 

12 

1 

6 

6 

0 

Total 

31 

24 

7 

8 

7 

1 

Total 

373 

218 

155 

131 

70 

61 

Chapter  5  Performance  Assessment 


55 


Table  22 

Breakdown  of  Discrimination  -  Area  B  and  Area  C-  NRL 


Ground- 

truth 

Demonstrator  Classification  | 

Confidence 

Area  B 

Area  C  || 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

4 

4 

0 

6 

6 

0 

Low 

83 

42 

41 

65 

26 

39 

Medium 

17 

13 

4 

23 

20 

3 

Total 

104 

59 

45 

94 

52 

42 

Clutter 

High 

6 

6 

0 

2 

2 

0 

Low 

4 

3 

1 

3 

3 

0 

Medium 

4 

4 

0 

2 

2 

0 

Total 

14 

13 

1 

7 

7 

0 

Ordnance 

High 

4 

4 

0 

1 

1 

0 

Low 

7 

4 

3 

4 

2 

2 

Medium 

6 

5 

1 

1 

1 

0 

Total 

17 

13 

4 

6 

4 

2 

Total 

135 

85 

50 

107 

63 

44 

Table  23  .  ^  ^ 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A-  NRL  without 

20/40  mm 

Ground- 

truth 

Demonstrator  Classification  I 

Confidence 

QA  Ranae 

Area  A 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

25 

25 

0 

2 

2 

0 

Low 

345 

163 

182 

99 

46 

53 

Medium 

197 

66 

131 

62 

21 

41 

Total 

567 

254 

313 

163 

69 

94 

Clutter 

High 

9 

9 

0 

0 

0 

0 

Low 

16 

10 

6 

4 

4 

0 

Medium 

10 

10 

0 

2 

2 

0 

Total 

35 

29 

6 

6 

6 

0 

Ordnance 

High 

10 

10 

0 

2 

2 

0 

Low 

16 

13 

3 

5 

5 

0 

Medium 

11 

7 

4 

2 

1 

1 

Total 

37 

30 

7 

9 

8 

1 

Total 

639 

313 

326 

178 

83 

95 

56 
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Table  24 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  NRL  without 
20/40  mm 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Deciared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

5 

5 

0 

18 

18 

0 

Low 

163 

71 

92 

83 

46 

37 

Medium 

88 

28 

60 

47 

17 

30 

Total 

256 

104 

152 

148 

81 

67 

Clutter 

High 

5 

5 

0 

4 

4 

0 

Low 

9 

5 

4 

3 

1 

2 

Medium 

5 

5 

0 

3 

3 

0 

Total 

19 

15 

4 

10 

8 

2 

Ordnance 

High 

5 

5 

0 

3 

3 

0 

Low 

8 

6 

2 

3 

2 

1 

Medium 

7 

4 

3 

2 

2 

0 

Total 

20 

15 

5 

8 

7 

1 

Total 

295 

134 

161 

166 

96 

70 

Table  2! 
Breakdt 
(EM61) 

) 

5wn  of  Discrimination  -  QA  Range  and  Area  A-  Parsons 

EM  and  Flag 

Ground- 

truth 

Demonstrator  Classification 

Confidence 

QA  Range 

Area  A  1 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

789 

783 

6 

123 

122 

1 

High 

8 

1 

7 

1 

0 

1 

Low 

1 

0 

1 

1 

0 

1 

Medium 

2 

2 

0 

1 

1 

0 

Total 

800 

786 

14 

126 

123 

3 

Clutter 

Undeclared 

76 

76 

0 

17 

17 

0 

High 

1 

0 

1 

1 

0 

1 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

78 

77 

1 

18 

17 

1 

Ordnance 

Undeclared 

58 

58 

0 

12 

12 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

1 

1 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

60 

60 

0 

12 

12 

0 

Total 

938 

923 

15 

156 

152 

4 
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Table  26 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Parsons  (EM- 
61)  EM  and  FLAG 


Ground- 

truth 

Demonstrator  Classification  || 

Confidence 

Area  B 

Area  C  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

333 

328 

5 

333 

333 

0 

High 

4 

1 

3 

3 

0 

3 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

338 

330 

8 

336 

333 

3 

Clutter 

Undeclared 

35 

35 

0 

24 

24 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

36 

36 

0 

24 

24 

0 

Ordnance 

Undeclared 

31 

31 

0 

15 

15 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

1 

1 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

33 

33 

0 

15 

15 

0 

Total 

407 

399 

8 

375 

372 

3 

Table  27 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  Parsons 
EM-61  Digital 


Ground- 

Truth 

Demonstrator  Classification  I 

Confidence 

QA  Ranqe 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

249 

96 

153 

61 

18 

43 

Low 

109 

0 

109 

62 

0 

62 

Medium 

1,149 

1 

1,148 

258 

1 

257 

Total 

1,507 

97 

1,410 

381 

19 

362 

Clutter 

High 

33 

23 

10 

13 

10 

3 

Low 

1 

0 

1 

1 

0 

1 

Medium 

27 

0 

27 

3 

0 

3 

Total 

61 

23 

38 

17 

10 

7 

Ordnance 

High 

37 

26 

11 

8 

5 

3 

Low 

1 

0 

1 

1 

0 

1 

Medium 

14 

0 

14 

3 

0 

3 

Total 

52 

26 

26 

12 

5 

7 

Total 

1,620 

146 

1,474 

410 

34 

376 

58 
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Table  28 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Parsons  EM-61 
Digital 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

81 

44 

37 

107 

34 

73 

Low 

3 

0 

3 

44 

0 

44 

Medium 

458 

0 

458 

433 

0 

433 

Total 

542 

44 

498 

584 

34 

550 

Clutter 

High 

9 

3 

6 

11 

10 

1 

Low 

0 

0 

0 

0 

0 

0 

Medium 

17 

0 

17 

7 

0 

7 

Total 

26 

3 

23 

18 

10 

8 

Ordnance 

High 

18 

13 

5 

11 

8 

3 

Low 

0 

0 

0 

0 

0 

0 

Medium 

7 

0 

7 

4 

0 

4 

Total 

25 

13 

12 

15 

8 

7 

Total 

593 

60 

533 

617 

52 

565 

Table  29 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  Parsons 
TM-5  EMU 


Ground 

Truth 

Demonstrator  Classification 

Confidence 

QA  Range 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

135 

131 

4 

3 

3 

0 

High 

3 

1 

2 

0 

0 

0 

Total 

138 

132 

6 

3 

3 

0 

Clutter 

Undeclared 

36 

36 

0 

8 

8 

0 

High 

0 

0 

0 

0 

0 

0 

Totai 

36 

36 

0 

8 

8 

0 

Ordnance 

Undeclared 

20 

20 

0 

6 

6 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

20 

20 

0 

6 

6 

0 

Total 

194 

188 

6 

17 

17 

0 
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Table  30 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Parsons  TM-5 
EMU 

Ground- 

truth 

Demonstrator  Classification  1 

Confidence 

Area  B 

Area  C  1 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

112 

109 

3 

20 

19 

1 

High 

3 

1 

2 

0 

0 

0 

Total 

115 

110 

5 

20 

19 

1 

Clutter 

Undeclared 

25 

25 

0 

3 

3 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

25 

25 

0 

3 

3 

0 

Ordnance 

Undeclared 

13 

13 

0 

1 

1 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

13 

13 

0 

1 

1 

0 

Total 

153 

148 

5 

24 

23 

1 

Identification  Results 

The  UXO  identification  performance  of  each  demonstrator  is  based  on  the 
UXO  type  declarations  included  in  each  of  the  required  prioritized  dig  lists.  The 
results  presented  in  this  section  have  been  adjusted  to  account  for  UXO-related 
items  that  were  present  in  Areas  A,  B,  and  C  from  previous  demonstration  work. 
Figures  32  through  36  include  classification  matrices  that  detail  and  summarize 
the  identification  performance  of  each  demonstrator.  There  are  two  types  of 
classification  matrices  presented. 

The  first  type  of  classification  matrix  is  the  target  classification  matrix, 
which  shows  how  well  the  demonstrator  identified  specific  ordnance  types  that 
were  detected.  This  classification  matrix  includes  the  following  entries: 

a.  The  entry  below  the  demonstrator’s  name  indicates  the  demonstration 
Area.  Area  A  is  the  four  30-m  grid.  Area  B  is  the  1 -hectare  site,  and  Area 
C  is  the  six  30-m  grid. 

b.  The  classes  across  the  top  of  the  matrix  are  the  classification  types  of  the 
actual  target  classes.  A  list  of  these  types  was  given  to  the  demonstrators 
for  use  in  classifying  their  detected  ordnance.  A  clutter  class  is  listed  as  a 
separate  entry  and  corresponds  to  a  false  alarm  if  classified  as  a  UXO 
target  and  to  a  correct  discrimination  when  classified  as  non-UXO  by  the 
demonstrator. 

c.  The  classification  column  on  the  left  side  of  the  matrix  lists  the 
demonstrator’s  declaration  for  each  detected  UXO  item. 

d.  The  totals  in  the  right-hand  column  of  the  matrix  correspond  to  the  total 
number  of  items  declared  by  the  demonstrator  as  a  particular  class. 
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e.  The  “%  Classified”  row  indicates  the  percentage  of  detected  targets,  for  a 
given  target  class,  that  were  correctly  classified. 

f.  The  “%  Classified  by  Class”  indicates  the  percentage  of  detected  targets, 
for  the  given  classes  of  projectile,  mortar,  rocket,  and  practice  bomb,  that 
were  correctly  classified. 

g.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  Low/Med 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  low  or  medium  confidence. 

h.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  High 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  high  confidence.  This  classi¬ 
fication  error  carried  the  highest  cost  penalty  (equivalent  to  the  cost  of  a 
complete  resurvey  of  the  area). 

Note  that  only  one  demonstrator  classified  their  declarations  with  the  given 
target  classes. 

The  second  type  of  classification  matrix  is  the  aggregate  size  classification 
matrix.  This  matrix  includes  the  following  entries: 

a.  The  entry  below  the  demonstrator’s  name  indicates  the  demonstration 
Area. 

b.  The  classes  across  the  top  of  the  matrix  are  the  aggregate  size  classes  of 
the  items  emplaced.  The  targets  are  grouped  into  three  classes.  The  small 
target  class  contains  the  20-mm  projectile,  the  40-mm  projectile,  and  the 
BDU  3  practice  bomb.  The  medium  class  contains  the  60-mm  mortar,  the 
81 -mm  mortar,  the  2.25-in.  rocket,  the  2.75-in.  rocket  warhead,  the 
SMAW  rocket,  the  LAAW,  the  BDU  33  practice  bomb,  the  MK-3 
practice  bomb,  and  the  MK-106  practice  bomb.  The  large  class  contains 
the  5-in.  projectile;  the  105-imn  projectile,  the  MK-81  practice  bomb,  the 
MK-82  practice  bomb,  and  the  MK-83  practice  bomb.  A  clutter  class  is 
listed  as  a  separate  entry  and  corresponds  to  a  false  alarm  if  classified  as 
a  UXO  target  and  to  a  correct  discrimination  when  classified  as  non- 
UXO  by  the  demonstrator. 

c.  The  classification  column  on  the  left  side  of  the  matrix  lists  the  demon¬ 
strator’s  declaration  for  each  detected  UXO  item.  Again,  the  declarations 
are  grouped  into  three  aggregate  size  classes  as  previously  described  in 
the  target  classes. 

d.  The  totals  in  the  right-hand  column  of  the  matrix  correspond  to  the  total 
number  of  items  declared  by  the  demonstrator  as  a  particular  class. 

e.  The  “%  Classified”  row  indicates  the  percentage  of  detected  targets,  for  a 
given  aggregate  size  class,  that  were  correctly  classified. 

/  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  Low/Med 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  low  or  medium  confidence. 
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g.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  High 

Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  high  confidence.  Again,  this 
classification  error  carried  the  highest  cost  penalty  (equivalent  to  the  cost 
of  a  complete  resurvey  of  the  area). 

Note  that  only  two  demonstrators  classified  their  declarations  with  enough 
descriptive  information  to  allow  for  the  evaluation  of  aggregate  size 
classification. 

NAEVA  was  the  only  demonstrator  that  submitted  specific  target  identifica¬ 
tions.  However,  their  classification  results,  shown  in  Figures  32  through  34,  were 
very  poor.  Their  results  are  only  slightly  better  when  aggregated  by  size,  as 
shown  in  Figures  35.  Classification  matrices  aggregated  by  size  for  Geophex  are 
shown  in  Figures  36.  Geophex  correctly  classified  a  few  more  ordnance  by  size 
than  NAEVA,  but  also  classified  more  nonordnance  items  as  ordnance.  NAEVA 
and  Geophex  are  the  only  demonstrators  who  attempted  to  identiiy  ordnance  by 
size. 


ROC-Based  Performance  Assessment 

Assessment  of  detection,  discrimination,  and  faise  alarm  rate 
performance 

One  of  the  critical  evaluation  factors  for  this  demonstration  is  the  detection 
and  discrimination  performance  of  the  advanced  systems  as  a  function  of  the 
number  of  false  alarms.  The  metrics  used  to  quantify  this  performance  consist  of 
the  pseudo  ROC  curves,  the  single-point  P(det)/Total  FAR,  and  the  maximum 
achievable  P(det).  The  methods  used  to  estimate  these  metrics  from  the  priori¬ 
tized  dig  lists  are  described  in  detail  in  Chapter  4.  Briefly,  the  pseudo  ROC 
curve,  which  graphically  represents  the  target  deteetion  percentage  vs.  the  total 
number  of  false  alarms  (or  false  alarm  rate  in  number  of  false  alarms  per  hee- 
tare),  is  calculated  by  sequentially  moving  from  the  top  of  the  prioritized  dig  list 
(i.e.,’the  highest  confidence  UXO  target  declaration)  and  determining  if  each 
object  on  the  list  (whether  classified  as  target  or  clutter)  corresponds  to  an 
emplaced  target  location  (a  correct  detection)  or  not  (a  false  alarm).  The  single¬ 
point  P(det)/Total  FAR  performance  is  based  on  the  point  on  the  ROC  curve  that 
corresponds  to  the  demonstrator-specified  dig  point  on  the  prioritized  dig  list,  and 
the  maximum  achievable  P(det)  is  based  on  the  highest  point  on  the  ROC  curve. 
These  performance  metrics  are  presented  in  the  following  graphs.  The  single¬ 
point  P(det)/Total  FAR  rate  is  shown  as  a  triangle  on  the  ROC  curve  (stop-dig- 
point)  and  it  is  noted  by  “SD”  in  the  legend.  The  probability  of  correct 
discrimination,  P(disc),  was  plotted  on  the  signal  strength  ROC  curves. 


Interpreting  pseudo-ROC  curves 

There  are  several  points  to  keep  in  mind  when  interpreting  these  pseudo 
ROC  curves: 
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NAEVA 

Classification  Matrix 

Area  A 

Targ 

et  Classification  by  Size 

r— 1 

Demonstrator's  Classification 

small 

medium 

large 

Clutter 

Total 

small 

0 

0 

0 

7 

7 

medium 

0 

6 

0 

46 

52 

large 

0 

0 

0 

7 

7 

Unknown  ordnance 

0 

1 

0 

2 

_ 

3 

Non-ordnance  Low/Mcd 

1 

3 

0 

81 

85 

Non-ordnance  High 

1 

1 

0 

143 

_ 

145 

_ 

Total 

2 

11 

0 

286 

299 

299 

%  Classified 

0.00% 

54.55% 

0.00% 

%  of  Total  Detected 

Targets  Classified  as 
Non-ordnance  Low/Mcd 
Class  30.77% 

%  of  Total  Detected 
Targets  Classified  as 
Non-ordnance  High 
Class  15.38% 

NAEVA 

Classification  Matrix 

Area  B 

Targ 

et  Classification  by  Size 

— — 1 

Demonstrator’s  Classification 

small 

medium 

large 

Clutter 

Total 

small 

1 

2 

0 

13 

_ 

16 

medium 

0 

6 

3 
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0 
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Targets  Classified  as 
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Class  10.00% 

Figure  35.  NAEVA  EM-63  size  classification  matrix  within  0.5  m 
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Geophex 

Classification  Matrix 
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small 
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55 
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0 

0 

0 

0 
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0 

0 
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0 
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2 
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1 
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0 

0 
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0 
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0 

0 

0 

0 
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5 

20 

8 
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45.00% 
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Non-ordnance  Low/Med 
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1 

56 
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0 

0 

0 

0 
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0 

0 

0 

26 

26 
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0 

0 

0 

0 

0 
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1 
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1 
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%  of  total  Detected 
Targets  Classified  as 
Non-ordnance  Low/Med 
Class _ 0.00%  I 


%  of  Total  Detected 
Targets  Classified  as 
Non-ordnance  High 
Class  0.00% 


%  of  Total  Detected 
Targets  Classified  as 
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Class  0.00% 


%  of  Total  Detected 
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Non-ordnance  High 
Class  0.00% 


Figure  36.  Geophex,  Ltd.  GEM-3  size  classification  matrix  within  0.5  m 
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a.  Abscissa  in  the  pseudo  ROC  curves  is  not  probability  of  false  alarms  but 
rather  total  number  of  false  alarms  in  a  hectare  or,  equivalently,  Total 
FAR  (number  of  false  alarms  per  hectare).  Therefore,  since  each  of  the 
Demonstration  Areas  A,  B,  and  C  are  different  sizes,  the  Total  FARs  are 
adjusted  to  the  ratio  of  false  alarms  per  hectare  so  that  rate  comparisons 
across  different  demonstration  areas  can  be  made. 

b.  These  curves  combine  detection  and  discrimination  of  ordnance  from 
nonordnance.  Thus,  the  initial  pseudo  ROC  curve  s  slope  represents  the 
anomalies  that  the  demonstrator  has  declared  as  UXO  with  the  highest 
confidence;  a  flat  slope  in  this  area  would  indicate  very  poor 
discrimination  capability. 

c.  Similarly,  the  final  slope  of  the  pseudo  curve  represents  anomalies  that 
the  demonstrator  has  declared  as  clutter  with  high  confidence;  a  positive 
slope  in  this  area  indicates  that  there  are  UXO  targets  which  the 
demonstrator  would  leave  unexcavated. 

Figures  37  through  51  show  the  results  obtained  when  the  location  accuracy 
is  set  to  0.5  m.  In  Appendix  C  (Figures  C5  through  C20),  the  analysis  is  per¬ 
formed  with  the  location  accuracy  set  to  1.0  m.  The  ROC-based  performance  of 
all  of  the  demonstrators  in  all  cases  fell  below  the  Kaho’olawe  Tier  II  Clearance 
requirements.  In  Figure  37,  Geophex  demonstrated  significantly  better  ROC 
performance  than  the  other  demonstrators,  and  GTL  demonstrated  significantly 
poorer  performance  (see  Figures  39, 45, 48,  and  51). 

Figures  37,  38,  and  39  show  the  demonstrators’  detection  performance  for 
Areas  A,  B,  and  C,  respectively.  The  distance  threshold  for  scoring  a  detection  is 
set  to  0.5  m  for  these  sets  of  plots.  Figure  37  shows  that  in  Area  A,  most  of  the 
systems  operated  along  very  similar  ROC  curves,  with  the  major  difference  being 
their  selection  of  the  “stop  dig”  or  end  point  threshold.  In  the  cases  of  NRL  and 
Parsons  TM-5  EMU,  the  endpoint  thresholds  were  set  so  high  that  the  operating 
(and  maximum  achievable)  P(det)  was  much  lower  than  those  of  the  other 
demonstrators.  Geophex  achieved  the  best  performance  for  this  area;  further¬ 
more,  it  is  apparent  from  the  steep  slope  of  the  ROC  curve,  that  a  lower  endpoint 
threshold  would  probably  have  resulted  in  increased  P(det)  with  a  relatively 
small  increase  in  total  FAR. 

The  placement  of  the  “stop  dig”  point,  (shown  as  the  “SD”  triangle  in  each  of 
these  figures)  indicates  the  demonstrators’  general  lack  of  confidence  in  their 
discrimination  capability.  With  the  exception  of  NAEVA,  all  other  demonstrators 
placed  their  “stop  dig”  point  at  the  end  of  the  dig  list.  NAEVA’s  attempt  at 
discrimination  resulted  in  an  operating  P(det)  that  was  10  percent  lower  than  the 
maximum  achievable  in  Area  A. 

Figure  38  shows  that  all  demonstrators,  with  the  exception  of  GTL,  achieved 
very  similar  ROC-based  performance  at  Area  B.  GTL  demonstrated  significantly 
poorer  detection  and  total  FAR  performance  than  the  others.  Overall,  all  of  the 
systems  demonstrated  poorer  ROC-based  performance  in  Area  B  than  Area  A, 
confirming  previous  reports  that  Area  B  contained  significantly  higher  levels  of 
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Pdet 


Area  A  by  Demonstrators 
within  0.5  m 


Figure  37.  Area  A  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  38.  Area  B  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  39.  Area  C  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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geologic  anomalies  and  metallic  clutter.  As  in  Area  A,  all  demonstrators  with  the 
exception  of  NAEVA,  were  extremely  conservative  in  their  “stop  dig”  point 
selection. 

Figure  39  shows  that  all  of  the  demonstrators  operated  along  very  similar 
ROC  curves  within  Area  C.  Again,  all  of  the  threshold  selections  were  very 
conservative.  With  the  exception  of  Geophex  and  NRL,  all  other  demonstrators 
performed  better  in  area  C  than  B,  but  none  performed  better  than  in  Area  A. 

Figures  40  through  42  show  the  results  of  reanalyzing  the  data  to  include 
only  targets  larger  than  40  mm  projectiles.  Only  NRL  and  Geophex  performed 
this  analysis  and  resubmitted  revised  dig  lists. 

The  small  number  of  submissions  and  the  lack  of  such  data  from  Parsons 
precludes  any  comparisons  across  systems  or  with  a  baseline.  In  order  to 
overcome  this  deficiency,  the  demonstrators  dig  lists  that  contained  all  targets 
(including  20  and  40  mm)  were  evaluated  with  the  20/40  mm  targets  removed 
from  the  groundtruth.  No  detection,  false  alarm,  or  missed  target  was  assigned  to 
any  declaration  within  1.0  m  of  these  small  emplaced  targets.  Figures  43  through 
45  show  the  results  of  this  evaluation.  These  ROC  curves  show  a  slight  improve¬ 
ment  in  performance  for  all  demonstrators,  but  no  significant  change  was  shown 
in  ROC  curve  shape  or  in  the  relative  performance  between  demonstrators.  It  can 
be  concluded  that  all  systems  have  similar  capability  (or  lack  thereof)  to  detect 
the  smaller  targets  in  the  Kaho’olawe  environment. 

Figures  46  through  48  show  the  P(det)  performance  as  a  function  of  Pfp  for 
all  demonstrators,  where  Pfp  is  computed  as  the  ratio  of  the  number  of  emplaced 
clutter  items  included  on  the  dig  list  to  the  number  of  clutter  items  emplaced. 
This  metric  attempted  to  separate  the  effects  of  the  geology  and  unknown  metal 
clutter  from  those  from  known,  emplaced  clutter  items.  These  ROC  curves  show 
very  small  differences  between  demonstrators,  but  most  importantly,  the  almost 
consistently  flat,  diagonal  shape  of  the  curves  indicates  that  none  of  the  systems 
demonstrated  a  capability  to  discriminate  emplaced  UXO  from  emplaced 
metallic  clutter  in  this  environment. 

Figures  49  through  51  show  the  systems’  ability  to  discriminate  ordnance 
from  nonordnance  (both  geologic  and  metallic)  based  solely  on  signal  strength. 
The  ROC  curves  for  all  demonstrators  tend  to  fall  within  a  narrow  band,  and  do 
not  support  any  conclusions  regarding  the  various  systems’  discrimination 
capability. 


Technology  Comparison 

The  detection  performance  of  all  demonstrated  systems  was  considerably 
lower  than  expected  and  significantly  lower  than  those  demonstrated  during  prior 
field  demonstrations  such  as  JPG  Phases  II  through  IV  and  during  the  first  phase 
of  this  project.  None  of  the  systems  demonstrated  ability  to  discriminate 
ordnance  from  metallic  clutter  nor  to  identify  ordnance  by  size  or  type.  It  was  not 
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Figure  40.  Area  A  -  P(det)  vs.  total  FAR  within  0.5  m  for  demonstrators  without  (wo)  20/40  mm 
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Figure  41 .  Area  B  -  P(det)  vs.  total  FAR  within  0.5  m  for  demonstrators  wo  20/40  mm 
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Area  C  by  Demonstrators  wo  20/40  mm 
within  0.5  m 


Figure  42.  Area  C  -  P(det)  vs.  total  FAR  within  0.5  m  for  demonstrators  wo  20/40  mm 
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Figure  43.  Area  A  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  44.  Area  B  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Area  C  wo  20/40  mm  by  Demonstrators 
within  0.5  m 


Figure  45.  Area  C  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  46.  Area  A  -  P(det)  vs.  Pfp  within  0.5  m  for  all  demonstrators 
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Area  B  by  Demonstrators 
within  0.5  m 


Figure  47.  Area  B  -  P(ciet)  vs.  Pfp  within  0.5  m  for  all  demonstrators 
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Figure  48.  Area  C  -  P(det)  vs.  Pfp  within  0.5  m  for  ail  demonstrators 
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Pdisc 


Figure  49.  Area  A  -  Pdisc  (prioritized  by  signai  strength)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  50.  Area  B  -  Pdisc  (prioritized  by  signal  strength)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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Figure  51 .  Area  C  -  Pdisc  (prioritized  by  signal  strength)  vs.  total  FAR  within  0.5  m  for  all  demonstrators 
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possible  to  evaluate  the  systems’  ability  to  discriminate  ordnance  from  geologic 
anomalies,  because  the  demonstration  area  was  so  cluttered  with  unknown 
metallic  objects  that  the  effects  of  geology  could  not  be  reliably  separated.  Some 
obvious  reasons  for  the  decreased  overall  performance  include  the  facts  that, 
unlike  the  fairly  benign,  low  noise  enviromnent  of  test  sites  such  as  JPG, 
Kaho’olawe  presents  an  extreme  in  clutter  density,  geologic  noise,  and  difficult 
operating  environment. 

Overall,  the  best  performances  were  achieved  in  Area  A  where  NAEVA, 
GTL,  Geophex,  Parson  EM  and  flag,  and  Parson  EM-61  digital  had  a  maximum 
achievable  P(det)  of  at  least  50  percent  when  including  detections  within  0.5  m. 
Also,  the  Total  FAR  was  the  lowest  for  all  demonstrators  in  Area  A.  However, 
none  of  the  demonstrators  reached  the  P(det)  required  for  Kaho’olawe  Tier  II 
clearance  (85  percent).  The  P(det)  in  Areas  B  and  C  were  significantly  lower  for 
all  demonstrators  and  their  Total  FAR  was  higher.  Most  significantly,  none  of  the 
advanced  EMI  systems  demonstrated  significant  improved  capability  over  the 
baseline  EM-61  system  operated  in  the  “EM  and  flag”  mode. 
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Field  Cost  Performance 

Labor  costs  associated  with  each  field  task  were  computed  by  applying  the 
cost  factors  described  in  the  Demonstration  Work  Plan  and  detailed  in  Table  31. 

It  should  also  be  noted  that  since  all  of  the  demonstrated  systems  were  man- 
portable  or  handheld,  with  similar  support  equipment  and  capital  cost  require¬ 
ments,  it  was  assumed  that  mobilization/demobilization  and  life  cycle  costs 
would  be  equal  and  could  be  omitted  from  this  relative  cost  performance 
evaluation. 

Analysis  of  this  table  indicates  that  the  field  labor  costs  of  most  of  the  dem¬ 
onstrators  fall  within  a  fairly  narrow  range,  with  the  two  exceptions  being  the 
Parsons  TM-5  EMU,  which  demonstrated  significantly,  lower  costs  and  NAEVA, 
which  achieved  significantly  higher  costs.  These  differences  may  be  attributable 
to  the  fact  that  the  TM-5EMU  survey  conducted  by  Parsons  did  not  appear  to 
adequately  cover  the  required  areas  (based  on  their  very  low  number  of  detected 
anomalies  and  on  the  fact  that  GTL,  using  the  same  sensor,  required  significantly 
more  time  to  complete  the  surveys).  A  further  reason  for  these  differences  could 
be  the  fact  that  NAEVA  approached  this  field  demonstration  effort  from  a  more 
scientific  perspective  where  one  of  their  primary  objectives  was  to  collect  very 
high  quality  field  data  to  support  their  ongoing  algorithm  development  efforts. 
With  the  exception  of  NAEVA,  all  of  the  other  advanced  EMI  systems  compare 
favorably  against  the  EM-61  “EM  and  flag”  baseline  technology. 


Weighted  Field  Cost  Performance 

Table  32  summarizes  the  operational  costs  of  the  demonstrator  systems  after 
the  cost  penalties  described  in  Chapter  4  were  applied.  These  penalties  consisted 
of  $200  for  each  false  alarm  (clutter  item  selected  for  digging  by  the  demon¬ 
strator)  and  the  cost  of  a  complete  resurvey  for  one  or  more  UXO  targets  missed 
(not  included  in  the  dig  list)  or  erroneously  declared  as  clutter  with  high  confi¬ 
dence.  This  table  highlights  the  fact  that  false  alarms  have  (by  a  large  margin)  the 
greatest  impact  on  the  cost  performance  of  each  system.  Table  32  indicates  that 
all  seven  demonstrators  were  penalized  with  the  cost  of  a  resurvey  at  each  of  the 
three  demonstration  areas  because  their  dig  lists  indicate  that  UXO  would  have 
had  been  left  in  the  ground.  It  is  difficult  to  draw  conclusions  regarding  the  per¬ 
formance  of  each  system  based  solely  on  these  results.  For  example,  one  may 
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Table  31 

Breakdown  of  Field  Costs 

Demonstrator 

Number  of 
Operators 

Categories 

Cost 

Time 
(hrs,  min) 

Cost  to  job 

NAEVA 

1 

Supervisor 

$95.00 

35:45 

$3,396.25 

3 

Logistic/Field  Setup 

28.50 

15:30 

441.75 

3 

Logistic/Field  Survey 

28.50 

83:00 

2,365.50 

3 

Logistic/Field  Downtime 

28.50 

1:30 

42.75 

3 

Logistic/Field  Resurvey 

28.50 

6:30 

185.25 

4 

Total 

$6,431.50 

GTL 

1 

Supervisor 

$95.00 

21:40 

$2,058.33 

1 

Logistic/Field  Setup 

28.50 

3:45 

106.88 

1 

Logistic/Field  Survey 

28.50 

13:43 

390.92 

1 

Logistic/Fietd  Downtime 

28.50 

3:34 

101.65 

1 

Logistic/Field  Resurvey 

28.50 

2 

Total 

$2,657.78 

Geophex 

1 

Supervisor 

$95.00 

24:30 

$2,327.50 

2 

Logistic/Field  Setup 

28.50 

10:30 

299.25 

2 

Logistic/Field  Survey 

28.50 

24:25 

688.75 

2 

Logistic/Field  Downtime 

28.50 

2:20 

66.50 

2 

Logistic/Field  Resurvey 

28.50 

2:55 

83.12 

3 

Total 

$3,465.12 

NRL 

1 

Supervisor 

$95.00 

20:02 

$1,903.17 

3 

Logistic/Field  Setup 

28.50 

9:00 

256.50 

3 

Logistic/Field  Survey 

28.50 

34:06 

977.55 

3 

Logistic/Field  Downtime 

28.50 

19:00 

541.50 

3 

Logistic/Field  Resurvey 

28.50 

4 

Total 

$3,678.72 

Parson  (EM-61) 
EM  and  Flag 

1 

Supervisor 

$95.00 

28:01 

$2,661.58 

2 

Logistic/Field  Setup 

28.50 

18:57 

540.08 

2 

Logistic/Field  Survey 

28.50 

58:30 

1,667.25 

2 

Logistic/Field  Downtime 

28.50 

17:00 

484.50 

2 

Logistic/Field  Resurvey 

28.50 

3 

Total 

$5,353.41 

Parson  EM-61 
Digital 

1 

Supervisor 

$95.00 

17:55 

$1,702.08 

1 

Logistic/Field  Setup 

28.50 

16:44 

476.60 

1 

Logistic/Field  Survey 

28.50 

15:04 

446.50 

1 

Logistic/Field  Downtime 

28.50 

7:16 

207.10 

1 

Logistic/Field  Resurvey 

28.50 

2:50 

80.75 

2 

Total 

$2,913.33 

Parson  TM-5 
EMU 

1 

Supervisor 

$95.00 

6:56 

$  658.67 

3 

Logistic/Field  Setup 

28.50 

12:10 

345.75 

3 

Logistic/Field  Survey 

28.50 

5:00 

142.50 

3 

Logistic/Field  Downtime 

28.50 

5:30 

156.75 

3 

Logistic/Field  Resurvey 

28.50 

4 

Total 

$1,304.67 
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Table  32 

Demonstrator  Costs  Including  Penalties  for  False  Alarms  and  for 
Leavinq  UXO  Targets  in  Ground 

NAEVA 

GTL 

Geophex 

NRL 

Parsons 

(EM-61) 

(EM  and 
Flag 

Parsons 
TM-5  EMU 

Parsons 

Digital 

Cost  to 
Survey 

$  2,366 

$  391 

$  689 

$  978 

$  1,667 

$  143 

$  447 

Cost  of 
Resurvev 

2,366 

391 

689 

978 

1,667 

143 

447 

Cost  of 

False 

Alarms 

128,000 

260,600 

157,800 

123,600 

176,200 

43,600 

319,400 

Total  Cost 

$132,732 

$261,382 

$159,178 

$125,556 

$179,534 

$43,886 

$320,294 

conclude  that  the  Parson  TM-5  EMU  system  is  superior  because  of  its  signifi¬ 
cantly  lower  eosts,  but  analysis  of  their  detection  performance  would  show  that  it 
failed  to  detect  a  very  large  percentage  of  the  UXO  present.  Since  the  cost  of 
false  alarms  dominates  this  type  of  analysis  in  very  highly  cluttered  areas  such  as 
Kaho’olawe,  any  system  eould  achieve  low  costs  by  operating  on  a  very  low 
point  on  the  ROC  curve.  Thus,  any  cost  comparisons  between  systems  have  to  be 
eorrelated  with  their  respective  ROC  curve  performance  in  order  to  reach 
reasonable  conclusions.  Assuming  that  the  systems  are  operated  at  reasonable 
points  of  the  ROC  curve,  Table  31  would  indicate  that  Parsons  EM-61  EM  and 
flag  and  NAEVA  are  the  most  cost-effective  performers.  Another  factor  to  be 
eonsidered  in  future  evaluations  is  the  fact  that  the  $200  cost  per  false  alarm  is 
excessive  for  sites  such  as  Kaho’olawe  where  the  high  density  and  fairly  shallow 
depths  of  most  metallic  clutter  from  munitions  fragments  allows  for  rapid 
removal  with  minimal  manpower  requirements. 
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7  Regulatory  Issues  - 
Compliance  and 
Acceptance 


Members  of  the  regulatory  community  who  are  aware  of  these  technology 
demonstrations  are  listed  in  Appendix  A. 
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8  Technology  Transition/ 
Implementation 


The  next  step  in  this  effort  eonsists  of  the  preparation  of  self-evaluation 
reports  by  eaeh  of  the  demonstrators  and  preparation  of  the  Project  Final  Report 
by  the  Government.  These  reports,  together  with  implementation  recommen¬ 
dations  will  be  forwarded  to  the  Navy  Facilities  Command  (NAVFAC)  and 
Parsons/UXB  personnel  currently  managing  and  conducting  UXO  remediation 
operations  at  Kaho’olawe.  Mr.  Jim  Putnam,  (808)  474-0559,  extension  224,  is  the 
project  manager  for  NAVFAC. 

The  planned  schedule  for  remaining  activities  under  this  ESTCP-funded 
project  are  as  follows: 


FY02  Milestones . Est.  Completion 

a.  Complete  Demonstrator  Self-Evaluation  Reports . 06/30/02 

b.  Complete  Project  Final  Report . 08/31/02 
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9  Lessons  Learned 


The  most  surprising  conclusion  from  this  demonstration  was  the  fact  that 
none  of  the  advanced  EMI  systems  demonstrated  significant  performance  and/or 
cost  improvements  over  the  baseline  technology  consisting  of  a  standard  EM-61 
system  operated  in  an  “EM  and  Flag”  mode.  The  relatively  good  performance 
achieved  by  this  system  in  this  mode  would  indicate  that  there  may  be  advan¬ 
tages  to  providing  real-time  feedback  to  the  UXO  survey  crews  so  that  they  may 
collect  additional  sensor  data  (in  orthogonal  directions)  over  suspected  anoma¬ 
lies,  rather  than  blindly  surveying  lanes  with  fixed  lane  widths  and  sampling 
rates.  Such  a  survey  method  would  also  allow  the  crew  to  visually  identity  and 
mark  surface  anomalies  that  could  otherwise  be  misinterpreted  as  UXO  during 
the  postsurvey  data  analysis. 

Another  important  lesson  learned  from  this  demonstration  is  the  difficulty  in 
setting  up  test  sites  and  conducting  demonstrations  at  live  UXO  sites  that  are  in 
the  process  of  being  remediated.  Even  though  the  calibration  and  demonstration 
areas  had  been  cleared  numerous  times,  there  were  still  excessive  amounts  of 
metallic  clutter  from  unknovra  sources  and  even  live  ordnance  that  remained  in 
these  areas.  As  a  result,  the  accuracy  of  the  groundtruth  available  for  such  test 
sites  is  always  in  doubt.  In  addition,  the  presence  of  unknown  metallic  clutter 
prevented  the  evaluation  of  the  advanced  EMI  systems’  assumed  capability  to 
mitigate  the  effects  of  geologic  noise.  Finally,  the  safety  and  logistics  problems 
associated  with  conducting  technology  demonstrations  concurrent  with  actual 
UXO  cleanup  operations  proved  to  be  a  very  inefficient,  costly,  and  time 
consuming  process. 

It  can  be  concluded  from  these  demonstrations  that  additional  research, 
development,  and  demonstration  work  is  needed  in  order  to  produce  UXO 
technologies  that  meet  reasonable  detection,  discrimination,  and  false  alarm 
performance  goals,  especially  in  difficult  sites  such  as  those  encountered  at 
Kaho’olawe. 


Chapter  9  Lessons  Learned 


91 


References 


Cespedes,  E.  R.  (2001).  “Advanced  UXO  detection/discrimination  technology 
demonstration  -  U.S.  Army  Jefferson  Proving  Ground,  Madison,  Indiana,” 
Technical  Report  ERDC/EL  TR-01-20,  U.S.  Army  Engineer  Research  and 
Development  Center,  Vicksburg,  MS. 

Geonics  Limited.  (1999).  “EM-63  full  time  domain  electromagnetic  UXO 
detector  operating  instructions,”  Ontario,  Canada. 

_ .  (2000).  “EM61  metal  detector  for  unexploded  ordnance  (UXO) 

detection  and  characterization,”  Ontario,  Canada. 

Geophysical  Technology  Limited.  (2002).  “Requested  information  regarding  the 
GTL  TM-5  EMU,”  Commercial-In-Confidence  correspondence  to  the 
U.S.  Army  Corps  of  Engineers. 

Naval  Research  Laboratory.  (2001).  “Man-portable  adjuncts  for  the  MTADS,” 
Technical  Report  NRL/PU/61 10-01-434,  Washington  DC. 

Naval  Explosive  Ordnance  Disposal  Technology  Division 

(NAVEODTECHDIV).  (2001a).  “Technology  demonstration  plan  - 
Advanced  UXO  detection/discrimination  technology  demonstration, 
Kaho’olawe  Island,  Hawaii,”  unpublished. 

_ .  (2001b).  “Site  preparation  plan  -  Advanced  UXO  detection/ 

discrimination  technology  demonstration,  Kaho’olawe  Island,  Hawaii,” 
unpublished. 

U.S.  Army  Environmental  Center  and  Naval  Explosive  Ordnance  Disposal 
Technology  Division.  (1997).  “UXO  technology  demonstration  program  at 
Jefferson  Proving  Ground,  Phase  III,”  Report  No.  SFIM-AEC-ET-CR-9701 1. 

Won,  1.  J.,  Keiswetter,  K.,  and  Novikova,  E.  (1998).  “Electromagnetic  induction 
spectroscopy,”  Journal  of  Environmental  and  Engineering  Geophysics  3(1), 
27-40. 


92 


References 


Appendix  A 
Points  of  Contact 


The  NAVEODTECHDIV  POC  is: 

Mr.  Hien  Dinh 

NAVEODTECHDIV 

2008  Stump  Neck  Road 

Indian  Head,  MD  20640-5070 

Phone:  (301)  7446850  ext.  267 

FAX:  (301)744-6947 

E-mail:  dinh@eodpoe2.navsea.navy.mil 

The  ERDC  POC  is: 

Dr.  Ernesto  Cespedes 

U.S.  Army  Engineer  Research  and  Development  Center 

3909  Halls  Ferry  Road 

Vicksburg,  MS  39180-6199 

Phone:  (601)  634-2655 

FAX:  (601)634-2732 

E-mail:  cespede@wes.army.mil 

The  AEC  POC  is: 

Mr.  George  Robitaille 
U.S.  Army  Environmental  Center 
ATTN:  SFIM-AEC-P2/ETD 
Bldg  E4430 

Aberdeen  Proving  Ground,  MD  21010-5401 
Phone:  (410)436-6850 
FAX:  (410)436-6836 

E-mail:  George.Robitaille@aec.apgea.army.mil 
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Important  points  of  contact  in  the  regulatory  and  user  community  who  have 
knowledge  of  the  demonstration  include: 

Interstate  Technology  Regulatory  Cooperation  (ITRC) 


ITRC  UXO  WORKGROUP 

Workgroup  Co-Leaders 

Jim  Austreng  (Team  Co-Lead) 

California  EPA 

Department  of  Toxic  Substances  Control 
10151  Croydon  Way,  Suite  3 
Sacramento,  CA  95827-2106 
P:  916-255-3702 
jaustren@dtsc .  ca.  gov 

Jennifer  Roberts  (Team  Co-Lead) 

Alaska  Department  of  Environmental  Conservation 
555  Cordova  Street 
Anchorage,  AK  99501 
P:  907-269-7553 

Jennifer_Roberts@envircon.state.ak.us 
Workgroup  Members 
David  Asiello 

U.S.  Department  of  Defense 
3400  Defense  Pentagon 
Washington,  DC  20301-3400 

P;  703-697-7363 . 

asielldj@acq.osd.mil 

Tim  Bahr 

Florida  Department  of  Environmental  Protection 
2600  Blair  Stone  Road 
Tallahassee,  FL  32399-2400 

P:  850-921-9984 . 

Tim.Bahr@dep.state.fl.us 

Geoff  Cullison 

Office  of  the  Chief  of  Naval  Operations 
Navy  Environmental  Restoration  Program 
2211  South  Clark  Street 
Arlington,  VA  22202-3735 

P:  703-602-5329 . 

cullison.geoffrey@hq.navy.mil 


F:  703-695-4981 


F:  850-922-4939 


F:  703-602-2676 
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Jeff  Edson 

Colorado  Department  of  Public  Health  &  Environment 
4300  Cherry  Creek  Drive,  South 

Denver,  CO  80246 

P:  303-692-3388 . 

Jeff.edson@state.co.us 

. F:  303-759-5355 

Dwight  Hempel 

Department  of  Interior 

Bureau  of  Land  Management 

1849  C  Street,  NW,  MS  lOOOLS 

Washington,  DC  20240 

P:  202-452-7778 . 

dwight_hempel@blm.gov 

. F:  202-452-7708 

Aimee  Houghton 

Center  for  Public  Environmental  Oversight 

122  C  Street  NW,  Suite  700 

Washington,  DC  20001-2109 

P;  202-662-1888 . 

aimeeh@cpeo.org 

. F:  202-628-1825 

Dave  Larsen 

Utah  Department  of  Environmental  Quality 

288  N.  1460  West 

Salt  Lake  City,  UT  84116 

P;  801-538-6749 . 

Dlarsen@deq.state.ut.us 

. F:  801-538-6715 

Mike  Liberati 

DuPont  Corporate  Remediation  Group 

2000  Cannonball  Road 

Pompton  Lakes,  NJ  07442 

P;  302-892-7421 

michael.r.liberati@usa.dupont.com 

Chris  Maurer 

Washington  Department  of  Ecology 

P.O.  Box  47600 

Olympia,  WA  98504 

P:  360-407-7223 

Cmau46 1  @ecy  .wa.gov 

Marshall  Nay 

TRW 

6001  Indian  School  Road 

Albuquerque,  NM  87110 

P:  505-998-8359 . 

. F:  505-998-8125 

marshall.nay@trw.com 
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Eric  Noack 

Nevada  Division  of  Environmental  Protection 
333  Nye  Lane,  ES-1 1 1 
Carson  City,  NV  89710 
P:  775-687-4670  x3032 

enoack@ndep.carson-city.nv.us 
Steve  Nussbaum 

Illinois  Environmental  Protection  Agency 
1021  N.  Grand  Ave.  East 
Springfield,  IL  62794-9276 

P:  217-782-9803 . F:  217-524-3291 

epa4129@epa.state.il.us 


James  Hersey 

Joint  UXO  Coordination  Office 
Attn:  AMSEL-RD-UXO-CO 
10221  BurbeckRoad 
Ft.  Belvoir,  VA  22060-5806 

P:  703-704-2609 . F:  ”703  704-2074 

jhersey@nvl.army.mil 

Rodney  Sobin 

Virginia  Department  of  Environmental  Quality 
629  East  Main  Street 
Richmond,  VA  23233 

P:  804-698-4382 . 804-698-4264 

rsobin@deq.state.va.us 

Jerry  Stamps 
Environmental  Engineer 

South  Carolina  Department  of  Health  and  Environmental  Conservation 
2600  Bull  Street 
Columbia,  SC  29201 

P:  803-896-4285 . F:  803-896-4002 

stampsjm@columb34.dhec. state. sc.us 


Philip  Stroud 

Alabama  Department  of  Environmental  Management 
P.O.  Box  301463 
Montgomery,  AL  36130 

P:  334-271-7750 . 

pns@adem.state.al.us 


F:  334-279-3050 


Jeff  Swanson 

Colorado  Department  of  Health  and  the  Environment,  Hazardous  Materials 
Division 

4300  Cherry  Creek  Drive,  South 
Denver,  CO  80246-1530 

P:  303-692-3416 . F:  303-759-5355 

Jeffrey.swanson@state.co.us 
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Julie  Wanslow 

New  Mexico  Department  of  the  Environment 
Harold  S.  Runnels  Building 
1190  St.  Francis  Dr. 

Santa  Fe,NM  87505-4182 
P:  505-827-1536 

julie_wanslow@nmenv.state.nm.us 
Greg  Zalaskus 

New  Jersey  Department  of  Environmental  Protection 

P.O.  Box  028 

401  East  State  Street 

Trenton,  NJ  08625-0413 

P:  609-984-2065 . F:  609-633-1545 

gzalasku@dep.state.nj  .us 


Appendix  A  Points  of  Contact 


A5 


Appendix  B 
Data  Archiving 


This  appendix  contains  digital  archive  directory/subdirectory  information 
about  the  sensor  data,  maps,  and  results  by  each  demonstrator  for  the  Advanced 
UXO  Detection/Discrimination  Technology  Demonstration,  Kaho’olawe, 
Hawaii.  The  Technology  Demonstration  Plan,  the  Site  Preparation  Plan,  and  the 
Geophysical  Background  Survey  report  are  contained  in  the  root  directory  of  the 
digital  archive.  To  obtain  a  copy  of  this  archive,  a  written  request  must  be 
submitted  to  one  of  the  authors  of  this  report. 
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Directory  map  (continued) 
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Appendix  C 

Evaluation  of  Data  at  1.0  m 


This  Appendix  contains  the  same  analysis  done  in  the  report  using  a  radius  of 
1.0  m  instead  of  the  0.5  m. 


Performance  Data 

In  accordance  with  the  Demonstration  Plan,  each  of  the  demonstrators  was 
responsible  for  determining  the  best  method  of  employing  his  system  in  order  to: 

a.  Ensure  full  coverage  of  each  demonstration  area. 

b.  Collect  high-quality  sensor  data  to  support  detection  and  discrimination 
requirements. 

c.  Achieve  high  production  rates. 

d.  Minimize  man-hour  requirements  and  costs. 

All  demonstrators  were  able  to  complete  the  field  surveys  within  the  allotted  time 
periods.  There  was  a  wide  range  in  the  demonstrators’  survey  data  quality,  data 
density,  quality  of  analysis,  and  compliance  with  the  data  submission  require¬ 
ments  specified  in  the  Demonstration  Plan  (NAVEODTECHDIV  2001a).’  For 
example,  a  number  of  demonstrators  failed  to  include  required  dig  list  informa¬ 
tion  such  as  recommended  stop  dig  point,  appropriate  confidence  levels,  and 
signal  strength  levels,  and  most  demonstrators  failed  to  reanalyze  their  data  and 
prepare  dig  lists  that  excluded  the  small  20-  and  40-mm  targets.  This  lack  of 
adherence  to  the  requirements  of  the  Demonstration  Plan  has  made  the  inter¬ 
pretation  of  results  and  adequate  across-demonstrator  performance  comparisons 
very  difficult.  This  appendix  presents  a  summary  of  the  data  submitted  by  the 
demonstrators  and  the  Government’s  assessment  of  their  performance. 

It  should  be  noted  that  because  of  the  very  high  numbers  of  false  alarms 
submitted  by  all  demonstrators,  the  Government  was  not  able  to  fully  investigate 
the  sources  of  all  of  them.  Nevertheless,  during  April  2002,  NAVEODTECHDIV 
personnel  conducted  extensive  surveys  and  excavation  activities  in  the  calibra¬ 
tion  and  demonstration  areas  in  order  to  verify  the  emplaced  target  locations  and 


’  Reference  information  follows  the  main  text. 
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to  attempt  to  identify  the  sourees  of  a  large  percentage  of  the  false  alarms. 
Information  from  these  postdemonstration  activities  was  incorporated  into  the 
groundtruth  data  used  to  evaluate  the  demonstrators’  performance. 


Detection  results 

Table  Cl  summarizes  the  number  of  UXO  targets  detected  within  a  1.0-m 
circular  error  by  each  demonstrator.  The  maximum  achievable  probability  of 
detection,  P(det),  is  also  given  in  the  table.  The  P(det)  is  calculated  as  the  number 
of  declared  items  that  correspond  to  emplaced  UXO  targets  (even  though  they 
may  have  been  misclassified  as  clutter)  divided  by  the  actual  number  of  UXO 
targets  emplaced  in  the  demonstration  site.  Table  C2  summarizes  the  detection 
results  achieved  when  the  small  UXO  targets  (20-  and  40-mm  projectiles)  were 
excluded  from  the  evaluation. 


Table  C1 

P(det)  bv  Area  within  1.0  m 

Within  1.0  m 

Total 

Area  A 

Area  B 

Area  C 

Number  of  Actual 
Targets 

24 

81 

34 

139 

NAEVA 

Targets  Detected 

18 

45 

20 

83 

P(det) 

0.750 

0.556 

0.588 

0.597 

GTL 

Targets  Detected 

18 

41 

18 

77 

P(det) 

0.750 

0.506 

0.529 

0.554 

Geophex 

Targets  Detected 

20 

46 

15 

81 

P(det) 

0.833 

0.568 

0.441 

0.583 

NRL 

Targets  Detected 

10 

23 

8 

37 

P(det) 

0.417 

0.284 

0.235 

0.266 

NRL  without  20/40  mm 

Targets  Detected 

11 

33 

12 

56 

P(det) 

0.458 

0.407 

0.353 

0.403 

Parsons  El\/I-61 

EM  and  Flag 

Targets  Detected 

18 

50 

22 

90 

P(det) 

0.750 

0.617 

0.647 

0.647 

Parsons  EM-61  Digital 

Targets  Detected 

18 

49 

23 

90 

P(det) 

0.750 

0.605 

0.676 

0.647 

Parsons  TM-5  EMU 

Targets  Detected 

7 

22 

4 

33 

P(det) 

0.292 

0.272 

0.118 

0.237 

These  results  indicate  that  no  demonstrator  was  able  to  achieve  the 
Kaho’olawe  Tier  II  clearance  requirements  of  0.85  P(det)  with  0.5-m  location 
accuracy  at  any  of  the  three  demonstration  areas.  Only  when  the  requirements 
were  relaxed  by  expanding  the  allowable  position  error  to  1.0  m  and  also  deleting 
the  smaller  UXO  targets  did  any  of  the  demonstrators  meet  the  P(det)  require¬ 
ments.  Even  then,  acceptable  P(det)  levels  were  only  obtained  in  Area  A.  Area  A 
had  considerably  lower  levels  of  geologic  noise  and  metallic  clutter  than  the 
other  two  areas. 
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Table  C2 

P(det)  by  Area  within  1.0  m  and  without  20/40  mm 


Within  1.0  m 

Totai 

Area  A 

Area  B  Area  C 

Number  of  Actual 
Targets  without 

20/40  mm 

19 

55 

28 

102 

NAEVA 

Targets  Detected 

16 

38 

19 

73 

P(det) 

0.842 

0.691 

0.679 

0.716 

GTL 

Targets  Detected 

17 

31 

15 

63 

P(det) 

0.895 

0.564 

0.536 

0.618 

Geophex 

Targets  Detected 

17 

39 

14 

70 

P(det) 

0.895 

0.709 

0.500 

0.686 

NRL 

Targets  Detected 

9 

22 

8 

37 

P(det) 

0.474 

0.400 

0.286 

0.363 

NRL  without  20/40  mm 

Targets  Detected 

10 

32 

11 

53 

P(det) 

0.526 

0.582 

0.393 

0.520 

Parsons  EM-61 

EM  and  Flag 

Targets  Detected 

17 

42 

21 

80 

P(det) 

0.895 

0.764 

0.750 

0.784 

Parsons  EM-61  Digital 

Targets  Detected 

16 

39 

22 

77 

P(det) 

0.842 

0.709 

0.786 

0.755 

Parsons  TM-5  EMU 

Targets  Detected 

7 

19 

4 

30 

P(det) 

0.368 

0.345 

0.143 

0.294 

Table  C3  shows  the  number  of  UXO  targets  detected  within  1.0  m  by  each 
demonstrator  in  each  30-  x  30-m  grid.  Shaded  entries  indicate  the  highest  number 
of  detections  for  that  grid.  The  last  column  of  the  table  indicates  the  number  of 
UXO  targets  not  detected  by  any  of  the  demonstrators.  Tables  4  through  1 1  show 
detections  by  ordnance  type  for  each  demonstrator.  Sixteen  different  ordnance 
types  are  listed  in  each  table.  No  UXO  targets  of  the  other  two  types  listed  in  the 
demonstration  plan  (BDU  3  practice  bomb  and  3-in.  projectile  practice  bomb) 
were  emplaced  for  this  demonstration. 


Discrimination  results 

Tables  C12  through  27  illustrate  the  comparison  of  the  demonstrators’ 
declarations  versus  the  ground-truth  emplaced  targets.  Table  CIS  summarizes  the 
performance  of  the  demonstrators  over  the  entire  demonstration  site  within  the 
QA  Range.  In  this  table,  the  ordnance  items  left  in  the  ground  represent  the  items 
declared  by  the  demonstrator  as  clutter  with  high  confidence  that  are  actually 
groundtruth  emplaced  ordnance  plus  the  number  of  undetected  ordnance  items. 
The  variable  Correct  Discrimination  is  the  total  of  the  correctly  identified 
ordnance  items.  The  False  Alarm  number  is  the  total  of  the  other  detections  plus 
the  total  of  the  Clutter  minus  the  number  of  objects  identified  as  Clutter  with 
High  confidence. 
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Table  C5 

UXO  Found  -  GTL 

Area  I 

Actual  Targets 

Within  1.0  m  I 

Description 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

2 

0 

2 

2.75-in. 

4 

5 

4 

13 

4 

0 

1 

5 

5-in. 

2 

4 

2 

8 

2 

3 

2 

7 

20-mm 

4 

12 

1 

17 

1 

3 

1 

5 

40-mm 

1 

14 

5 

20 

0 

9 

2 

11 

60-mm 

2 

11 

3 

16 

2 

5 

3 

10 

81 -mm 

1 

8 

3 

12 

1 

7 

1 

9 

105  mm 

1 

3 

1 

5 

1 

2 

1 

4 

BDU-33 

3 

4 

3 

10 

3 

2 

2 

7 

Mk-3 

1 

4 

3 

8 

1 

2 

1 

4 

Mk-81 

1 

2 

1 

4 

1 

0 

0 

1 

Mk-82 

1 

4 

7 

12 

1 

2 

4 

7 

Mk-83 

1 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

4 

0 

4 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

0 

0 

0 

0 

Total 

24 

81 

34 

139 

18 

41 

18 

77 

Table  C6 

UXO  Found  -  Geophex  &  Geophex  without  20/40  mm 


Description 

Area 

Actual  Targets 

Within  1.0  m  I 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

3 

0 

3 

2.75-in. 

4 

5 

4 

13 

4 

2 

1 

7 

5-in. 

2 

4 

2 

8 

1 

4 

2 

7 

20-mm 

4 

12 

1 

17 

2 

1 

0 

3 

40-mm 

1 

14 

5 

20 

1 

6 

1 

8 

60-mm 

2 

11 

3 

16 

2 

5 

2 

9 

81 -mm 

1 

8 

3 

12 

1 

8 

1 

10 

105-mm 

1 

3 

1 

5 

1 

3 

0 

4 

BDU-33 

3 

4 

3 

10 

3 

3 

2 

8 

Mk-3 

1 

4 

3 

8 

1 

4 

2 

7 

Mk-81 

1 

2 

1 

4 

1 

1 

1 

3 

Mk-82 

1 

4 

7 

12 

1 

2 

3 

6 

Mk-83 

1 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

4 

0 

4 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

1 

0 

0 

1 

Total 

24 

81 

34 

139 

20 

46 

15 

81 

Note:  Geophex  without  20/40  mm  detected  the  same  ordnance  items. 
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Table  C7 

UXO  Found  -  NRL 

Description 

Area  I 

Actual  Targets 

Within  1.0  m  II 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

2 

0 

2 

2.75-in. 

4 

5 

4 

13 

1 

1 

1 

3 

5-in. 

2 

4 

2 

8 

1 

2 

0 

3 

20-mm 

4 

12 

1 

17 

1 

0 

0 

1 

40-mm 

1 

14 

5 

20 

0 

1 

0 

1 

60-nrim 

2 

11 

3 

16 

1 

2 

1 

4 

81 -mm 

1 

8 

3 

12 

0 

4 

0 

4 

105-mm 

1 

3 

1 

5 

0 

2 

0 

2 

BDU-33 

3 

4 

3 

10 

3 

3 

1 

7 

Mk-3 

1 

4 

3 

8 

1 

2 

1 

4 

Mk-81 

1 

2 

1 

4 

1 

0 

1 

2 

Mk-82 

1 

4 

7 

12 

0 

2 

3 

5 

Mk-83 

1 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

2 

0 

2 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

0 

0 

0 

0 

Total 

24 

81 

34 

139 

10 

23 

8 

41 

Table  C8 

UXO  Found  -  NRL  without  20/40  mm 

Description 

Area  II 

Actual  Targets 

Within  1.0  m  II 

A 

B 

C 

Total 

A 

B 

c 

Total 

2.25-in. 

0 

3 

0 

3 

0 

3 

0 

3 

2.75-in. 

4 

5 

4 

13 

1 

1 

1 

3 

5-in. 

2 

4 

2 

8 

1 

3 

0 

4 

20-mm 

4 

12 

1 

17 

1 

0 

0 

1 

40-mm 

1 

14 

5 

20 

0 

1 

1 

2 

60-mm 

2 

11 

3 

16 

2 

4 

2 

8 

81-mm 

1 

8 

3 

12 

0 

5 

1 

6 

105-mm 

1 

3 

1 

5 

0 

3 

0 

3 

BDU-33 

3 

4 

3 

10 

3 

4 

1 

8 

Mk-3 

1 

4 

3 

8 

1 

4 

1 

6 

Mk-81 

1 

2 

1 

4 

1 

0 

1 

2 

Mk-82 

1 

4 

7 

12 

0 

2 

3 

5 

Mk-83 

1 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

3 

0 

3 

LAAW 

2 

1 

0 

3 

1 

0 

0 

1 

SMAW 

1 

1 

1 

3 

0 

0 

1 

1 

Total 

24 

81 

34 

139 

11 

33 

12 

56 
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Table  C9 

UXO  Found  -  Parsons  EM-61  EM  &  Flag 


Description 

Area  I 

Actual  Targets 

Within  1.0  m  I 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

3 

0 

3 

2.75-in. 

4 

5 

4 

13 

4 

3 

1 

8 

5-in. 

2 

4 

2 

8 

2 

4 

2 

8 

20-mm 

4 

12 

1 

17 

0 

1 

0 

1 

40-mm 

_J _ 

14 

5 

20 

1 

7 

1 

9 

60-mm 

2 

11 

3 

16 

1 

6 

3 

10 

81-mm 

1 

8 

3 

12 

1 

7 

2 

10 

105-mm 

1 

3 

1 

5 

1 

2 

1 

4 

BDU-33 

3 

4 

3 

10 

3 

4 

2 

9 

Mk-3 

1 

4 

3 

8 

1 

4 

3 

8 

Mk-81 

1 

2 

1 

4 

1 

2 

1 

4 

Mk-82 

1 

4 

7 

12 

1 

3 

5 

9 

Mk-83 

1 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

4 

0 

4 

LAAW 

2 

1 

0 

3 

2 

0 

0 

2 

SMAW 

1 

1 

1 

3 

0 

0 

1 

1 

Total 

24 

81 

34 

139 

18 

50 

22 

90 

Table  C10 

UXO  Found  -  Parsons  EM-61  Digital 


Description 

Area  I 

Actual  Targets 

Within  1.0  m  1 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

2 

0 

2 

2.75-in. 

4 

5 

4 

13 

4 

2 

2 

8 

5-in. 

2 

4 

2 

8 

2 

3 

1 

6 

20-mm 

4 

12 

1 

17 

1 

1 

0 

2 

40-mm 

1 

14 

5 

20 

1 

9 

1 

11 

60-mm 

2 

11 

3 

16 

2 

8 

3 

13 

81-mm 

1 

8 

3 

12 

1 

8 

2 

11 

105-mm 

1 

3 

1 

5 

1 

3 

1 

5 

BDU-33 

3 

4 

3 

10 

2 

3 

3 

8 

Mk-3 

1 

4 

3 

8 

1 

2 

3 

6 

Mk-81 

1 

2 

1 

4 

1 

1 

1 

3 

Mk-82 

1 

4 

7 

12 

0 

2 

5 

7 

Mk-83 

0 

1 

0 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

0 

4 

0 

4 

LAAW 

2 

1 

0 

3 

2 

1 

0 

3 

SMAW 

1 

1 

1 

3 

0 

0 

1 

1 

Total 

24 

81 

34 

139 

18 

49 

23 

90 
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Table  C11 

UXO  Found  -  Parsons  EM-61  TIVi-5  EMU 


Description 

Area  I 

Actual  Tarqets 

Within  1.0  m 

A 

B 

C 

Total 

A 

B 

C 

Total 

2.25-in. 

0 

3 

0 

3 

0 

1 

0 

1 

2.75-in. 

4 

5 

4 

13 

0 

3 

0 

3 

5-in. 

2 

4 

2 

8 

0 

1 

0 

1 

20-mm 

4 

12 

1 

17 

0 

0 

0 

0 

40-mm 

1 

14 

5 

20 

0 

3 

0 

3 

60-mm 

2 

11 

3 

16 

1 

4 

1 

6 

81-mm 

1 

8 

3 

12 

1 

1 

1 

3 

105-mm 

1 

3 

1 

5 

1 

1 

0 

2 

BDU-33 

3 

4 

3 

10 

0 

0 

0 

0 

Mk-3 

1 

4 

3 

8 

3 

2 

0 

5 

Mk-81 

1 

2 

1 

4 

0 

1 

0 

1 

Mk-82 

1 

4 

7 

12 

0 

1 

0 

1 

Mk-83 

0 

1 

0 

1 

0 

0 

0 

0 

Mk-106 

0 

4 

0 

4 

1 

1 

0 

2 

LAAW 

2 

1 

0 

3 

0 

3 

2 

5 

SMAW 

1 

1 

1 

3 

0 

0 

0 

0 

Total 

24 

81 

34 

139 

7 

22 

4 

33 

Table  13  through  27  show  more  detailed  breakdowns  of  the  results  by  area 
for  each  demonstrator.  The  left  most  column  of  each  of  these  tables  represents 
the  emplaced  targets.  These  items  are  matched  with  the  demonstrators’  dig  list. 
When  the  demonstrator  reported  the  object  as  ordnance  and  the  emplaced  target 
was  a  groundtruth  ordnance  match  then  it  is  considered  a  correct  discrimination. 
The  confidence  of  the  demonstrators’  declaration  is  used  to  determine  which 
items  from  his  dig  list  are  to  be  excavated.  In  lieu  of  a  demonstrators’  defined 
“stop  dig  point,”  only  objects  that  are  declared  as  clutter  with  high  confidence 
would  be  left  in  the  ground. 

For  example,  NAEVA  (Under  1.0  m)  discrimination  performance  is 
illustrated  in  Table  C13.  NAEVA  reported  295  items  as  Ordnance.  Of  that 
number,  51  actually  matched  the  ground-truth  ordnance  items  emplaced  in  the 
ground,  39  matched  the  ground-truth  emplaced  fragments,  and  205  detected 
something  other  than  the  targets  emplaced  in  the  ground.  NAEVA  reported 
630  items  as  clutter.  Only  37  actually  matched  the  ground-truth  fragments 
emplaced  in  the  ground,  32  matched  the  ground-truth  emplaced  ordnance  items, 
and  561  detected  something  other  than  the  targets  emplaced  in  the  ground. 
NAEVA  reported  250  (240+4+6)  with  high  confidence  of  the  630  clutter  items, 
therefore,  40  percent  of  the  identified  clutter  were  designated  as  “do  not  dig.” 
There  were  six  ordnance  items  that  would  have  been  left  in  the  ground  as  a  result 
of  being  classified  as  high  confidence  clutter.  Also,  from  Table  C12,  there  were 
62  undetected  ordnance  items.  NAEVA  reported  a  total  of  766  items  which  did 
not  match  any  emplaced  ordnance,  and  76  items  that  matched  emplaced 
fragments. 
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Tabled  2 

Summary  of  Discrimination  Performance 

Ordnance  Left  in  the 
Ground 

Correct  Discrimination 

Raise 

Aiarm 

Number 

Within  1.0  m 

Within  1.0  m 

Within 

1.0  m 

Num 

Percent 

Num 

Percent 

Naeva 

62 

44.60 

77 

64.03 

842 

GTL 

62 

45.32 

77 

55.40 

1,249 

Geophex 

58 

41.73 

81 

58.27 

750 

Geophex  without 

20/40  mm 

58 

41.73 

81 

58.27 

750 

NRL 

98 

70.50 

41 

29.50 

333 

NRL  without 

20/40  mm 

83 

59.71 

56 

40.29 

584 

49 

35.25 

90 

64.75 

854 

Parson  EM-61  Digital 

49 

35.25 

90 

64.75 

1,530 

Parson  TM-5  EMU 

106 

76.26 

33 

23.74 

167 

Table  C13 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  NAEVA 


Demonstrator  Classification  |j 

Confidence 

QA  Range 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

366 

126 

240 

168 

28 

140 

Low 

400 

79 

321 

94 

22 

72 

Total 

766 

205 

561 

262 

50 

212 

Clutter 

High 

33 

29 

4 

10 

7 

3 

Low 

43 

10 

33 

9 

2 

7 

Total 

76 

39 

37 

19 

9 

10 

Ordnance 

High 

43 

37 

6 

10 

8 

2 

Low 

40 

14 

26 

8 

2 

6 

Total 

83 

51 

32 

18 

10 

8 

Total 

925 

295 

630 

299 

69 

230 
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Table  C14 

Breakdown  of  Discrimination  —  Area  B  and  Area  C  —  NAEVA 


Ground- 

truth 

uemonsiraio 

Confidence 

Area  B 

Area  C  1 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalie 

s 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

121 

57 

64 

77 

41 

36 

Low 

173 

31 

142 

133 

26 

107 

Total 

294 

88 

206 

210 

67 

143 

Clutter 

High 

15 

14 

1 

8 

8 

0 

Low 

18 

3 

15 

16 

5 

11 

Total 

33 

17 

16 

24 

13 

11 

Ordnance 

High 

22 

19 

3 

11 

10 

1 

Low 

23 

8 

15 

9 

4 

5 

Total 

45 

27 

18 

20 

14 

6 

Total 

372 

132 

240 

254 

94 

160 

Tabled  5 

Breakdown  of  Discrimination  -  QA  Range  -  GTL 


Ground- 

truth 

Demonstrator  Classification  1 

Confidence 

Total 

Area  A 

Area  B 

Area  C 

Declared 

Ordnance 

Declared 

Ordnance 

Declared 

Ordnance 

Declared 

Ordnance 

Non- 

emplaced 

High 

649 

90 

315 

244 

Low 

293 

65 

158 

70 

Medium 

225 

52 

117 

56 

Total 

1,167 

207 

590 

370 

Clutter 

High 

58 

15 

28 

15 

Low 

15 

2 

9 

4 

Medium 

9 

4 

4 

1 

Total 

82 

21 

41 

20 

Ordnance 

High 

56 

12 

29 

15 

Low 

10 

2 

5 

3 

Medium 

11 

4 

7 

0 

Total 

77 

18 

41 

18 

Total 

1,326 

246 

672 

408 

Note:  GTL  declared  all  items  on  its  list  as  Ordnance. 
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Table  C16 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  Geophex  & 
Geophex  without  20/40  mm  _ 


Ground- 

truth 

Demonstrator  Classification  I 

Confidence 

QA  Range 

Area  A  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

93 

89 

4 

15 

15 

0 

Low 

276 

236 

40 

51 

46 

5 

Medium 

296 

263 

33 

42 

39 

3 

Total 

665 

588 

77 

108 

100 

8 

Clutter 

High 

26 

26 

0 

10 

10 

0 

Low 

22 

18 

4 

3 

2 

1 

Medium 

41 

37 

4 

9 

9 

0 

Total 

89 

81 

8 

22 

21 

1 

Ordnance 

High 

25 

25 

0 

6 

6 

0 

Low 

22 

21 

1 

5 

5 

0 

Medium 

34 

33 

1 

9 

8 

1 

Total 

81 

79 

2 

20 

19 

1 

Total 

835 

748 

87 

150 

140 

10 

Table  C17 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Geophex  & 
Geophex  without  20/40  mm  _ 


Ground- 

truth 

Demonstrator  Classification  I 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

60 

57 

3 

18 

17 

1 

Low 

166 

147 

19 

59 

43 

16 

Medium 

189 

166 

23 

65 

58 

7 

Total 

415 

370 

45 

142 

118 

24 

Clutter 

High 

12 

12 

0 

4 

4 

0 

Low 

13 

11 

2 

6 

5 

1 

Medium 

21 

17 

4 

11 

11 

0 

Total 

46 

40 

6 

21 

20 

1 

Ordnances 

High 

14 

14 

0 

5 

5 

0 

Low 

13 

13 

0 

4 

3 

1 

Medium 

19 

19 

0 

6 

6 

0 

Total 

46 

46 

0 

15 

14 

1 

Total 

507 

456 

51 

178 

152 

26 
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Tabled  8 

Breakdown  of  Discrimination  —  QA  Range  and  Area  A  —  NRL 


Ground- 

truth 

Demonstrator  Classification  II 

Confidence 

QA  Ranae 

Area  A 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

6 

6 

0 

0 

0 

0 

Low 

215 

87 

128 

78 

27 

51 

Medium 

71 

55 

16 

32 

23 

9 

Total 

292 

148 

144 

110 

50 

60 

Clutter 

High 

11 

11 

0 

0 

0 

0 

Low 

17 

14 

3 

5 

5 

0 

Medium 

12 

12 

0 

6 

6 

0 

Totai 

40 

37 

3 

11 

11 

0 

Ordnance 

High 

6 

6 

0 

0 

0 

0 

Low 

21 

14 

7 

4 

3 

1 

Medium 

14 

13 

1 

6 

6 

0 

Total 

41 

33 

8 

10 

9 

1 

Total 

373 

218 

155 

131 

70 

61 _ 

Table  C19 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  NRL 


Demonstrator  Classification 

Ground- 

truth 

Area  B 

Area  C 

Confidence 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

High 

2 

2 

0 

4 

4 

0 

emplaced 

Low 

75 

36 

39 

62 

24 

38 

Medium 

16 

12 

4 

23 

20 

3 

Total 

93 

50 

43 

89 

48 

41 

Clutter 

High 

7 

7 

0 

4 

4 

0 

Low 

8 

5 

3 

4 

4 

0 

Medium 

4 

4 

0 

2 

2 

0 

Totai 

19 

16 

3 

10 

10 

0 

Ordnance 

High 

5 

5 

0 

1 

1 

0 

Low 

11 

8 

3 

6 

3 

3 

Medium 

7 

6 

1 

1 

1 

0 

Total 

23 

19 

4 

8 

5 

3 

Total 

135 

85 

50 

107 

63 

44 
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Table  C20 

Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  NRL  without 
20/40  mm 


Ground- 

truth 

Demonstrator  Classification  | 

Confidence 

QA  Range 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

20 

20 

0 

0 

0 

0 

Low 

318 

146 

172 

93 

41 

52 

Medium 

188 

61 

127 

61 

20 

41 

Total 

526 

227 

299 

154 

61 

93 

Clutter 

High 

13 

13 

0 

2 

2 

0 

Low 

29 

17 

12 

7 

6 

1 

Medium 

14 

13 

1 

3 

3 

0 

Total 

56 

43 

13 

12 

11 

1 

Ordnance 

High 

11 

11 

0 

2 

2 

0 

Low 

30 

23 

7 

8 

8 

0 

Medium 

16 

9 

7 

2 

1 

1 

Total 

57 

43 

14 

12 

11 

1 

Total 

639 

313 

326 

178 

83 

95 

Table  C21 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  NRL  without 
20/40  mm 


Ground- 

truth 

Demonstrator  Classification  I 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

4 

4 

0 

16 

16 

0 

Low 

147 

64 

83 

78 

41 

37 

Medium 

82 

25 

57 

45 

16 

29 

Total 

233 

93 

140 

139 

73 

66 

Clutter 

High 

5 

5 

0 

6 

6 

0 

Low 

17 

8 

9 

5 

3 

2 

Medium 

7 

6 

1 

4 

4 

0 

Total 

29 

19 

10 

15 

13 

2 

Ordnance 

High 

6 

6 

0 

3 

3 

0 

Low 

16 

10 

6 

6 

5 

1 

Medium 

11 

6 

5 

3 

2 

1 

Total 

33 

22 

11 

12 

10 

2 

Total 

295 

134 

161 

166 

96 

70 
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Table  C22 

Breakdown  of  Discrimination  —  QA  Range  and  Area  A  —  Parsons 
(EM-61)  EM  and  Flag 


P' 

Ground- 

truth 

Demonstrator  Classification  II 

Confidence 

QA  Range 

Area  A  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

746 

740 

6 

114 

113 

1 

High 

8 

1 

7 

1 

0 

1 

Low 

1 

0 

1 

1 

0 

1 

Medium 

2 

2 

0 

1 

1 

0 

Total 

757 

743 

14 

117 

114 

3 

Clutter 

Undeclared 

89 

89 

0 

20 

20 

0 

High 

1 

0 

1 

1 

0 

1 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

91 

90 

1 

21 

20 

1 

Ordnance 

Undeclared 

88 

88 

0 

18 

18 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

1 

1 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

90 

90 

0 

18 

18 

0 

Total 

938 

923 

15 

156 

152 

4 

Table  C23 

Breakdown  of  Discrimination  -  Area  B  -  Parsons  (EM-61)  EM  and 
Flag 


Ground- 

truth 

Demonstrator  Classification  II 

Confidence 

Area  B 

Area  C  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

311 

306 

5 

321 

321 

0 

High 

4 

1 

3 

3 

0 

3 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

316 

308 

8 

324 

321 

3 

Clutter 

Undeclared 

40 

40 

0 

29 

29 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

0 

0 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

41 

41 

0 

29 

29 

0 

Ordnance 

Undeclared 

48 

48 

0 

22 

22 

0 

High 

0 

0 

0 

0 

0 

0 

Low 

1 

1 

0 

0 

0 

0 

Medium 

1 

1 

0 

0 

0 

0 

Total 

50 

50 

0 

22 

22 

0 

Total 

407 

399 

8 

375 

372 

3 
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Table  C24 


Breakdown  of  Discrimination  -  QA  Range  and  Area  A  -  Parsons 
EM-61  Digital 


Ground- 

truth 

Demonstrator  Classification  | 

Confidence 

QA  Range 

Area  A  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

221 

83 

138 

55 

16 

39 

Low 

107 

0 

107 

61 

0 

61 

Medium 

1,091 

1 

1,090 

252 

1 

251 

Total 

1,419 

84 

1,335 

368 

17 

351 

Clutter 

High 

49 

29 

20 

16 

11 

5 

Low 

3 

0 

3 

2 

0 

2 

Medium 

59 

0 

59 

6 

0 

6 

Total 

111 

29 

82 

24 

11 

13 

Ordnance 

High 

49 

33 

16 

11 

6 

5 

Low 

1 

0 

1 

1 

0 

1 

Medium 

40 

0 

40 

6 

0 

6 

Total 

90 

33 

57 

18 

6 

12 

Total 

1,620 

146 

1,474 

410 

34 

376 

Table  C25 

Breakdown  of  Discrimination  -  Area  B  and  Area  C  -  Parsons  EM-61 
Digital 


Ground- 

truth 

Demonstrator  Classification 

Confidence 

Area  B 

Area  C  I 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

High 

63 

34 

29 

103 

33 

70 

Low 

3 

0 

3 

43 

0 

43 

Medium 

423 

0 

423 

416 

0 

416 

Total 

489 

34 

455 

562 

33 

529 

Clutter 

High 

19 

8 

11 

14 

10 

4 

Low 

0 

0 

0 

1 

0 

1 

Medium 

36 

0 

36 

17 

0 

17 

Total 

55 

8 

47 

32 

10 

22 

Ordnance 

High 

26 

18 

8 

12 

9 

3 

Low 

0 

0 

0 

0 

0 

0 

Medium 

23 

0 

23 

11 

0 

11 

Total 

49 

18 

31 

23 

9 

14 

Total 

593 

60 

533 

617 

52 

565 
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Table  C26 

Breakdown  of  Discrimination  —  QA  Range  and  Area  A  —  Parsons 
TM-5  EMU  _ 


Ground- 

truth 

Demonstrator  Classification  II 

Confidence 

QA  Ranoe 

Area  A  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

120 

116 

4 

2 

2 

0 

High 

3 

1 

2 

0 

0 

0 

Total 

123 

117 

6 

2 

2 

0 

Clutter 

Undeclared 

38 

38 

0 

8 

8 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

38 

38 

0 

8 

8 

0 

Ordnance 

Undeclared 

33 

33 

0 

7 

7 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

33 

33 

0 

7 

7 

0 

Total 

194 

188 

6 

17 

17 

0 

Table  C27 

Breakdown  of  Discrimination  —  Area  B  and  Area  C  —  Parsons  TM-5 
EMU 

Ground- 

truth 

Demonstrator  Classification  II 

Confidence 

Area  B 

Area  C  II 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Total 

Anomalies 

Declared 

Ordnance 

Declared 

Clutter 

Non- 

emplaced 

Undeclared 

102 

99 

3 

16 

15 

1 

High 

3 

1 

2 

0 

0 

0 

Total 

105 

100 

5 

16 

15 

1 

Clutter 

Undeclared 

26 

26 

0 

4 

4 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

26 

26 

0 

4 

4 

0 

Ordnance 

Undeclared 

22 

22 

0 

4 

4 

0 

High 

0 

0 

0 

0 

0 

0 

Total 

22 

22 

0 

4 

4 

0 

Total 

153 

148 

5 

24 

23 

1 

Identification  results 

The  UXO  identification  performance  of  each  demonstrator  is  based  on  the 
UXO  type  declarations  included  in  each  of  the  required  prioritized  dig  lists.  The 
results  presented  in  this  section  have  been  adjusted  to  account  for  UXO-related 
items  that  were  present  in  Areas  A,  B,  and  C  from  previous  demonstration  work. 
Figures  Cl  through  C5  include  classification  matrices  that  detail  and  summarize 
the  identification  performance  of  each  demonstrator.  There  are  two  types  of 
classification  matrices  presented. 

The  first  type  of  classification  matrix  is  the  target  classification  matrix, 
which  shows  how  well  the  demonstrator  identified  specific  ordnance  types  that 
were  detected.  This  classification  matrix  includes  the  following  entries; 
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a.  The  entry  below  the  demonstrator’s  name  indicates  the  demonstration 
Area.  Area  A  is  the  four  30-m  grid,  Area  B  is  the  1 -hectare  site,  and  Area 
C  is  the  six  30-m  grid. 

b.  The  classes  across  the  top  of  the  matrix  are  the  classification  types  of  the 
actual  target  classes.  A  list  of  these  types  was  given  to  the  demonstrators 
for  use  in  classifying  their  detected  ordnance.  A  clutter  class  is  listed  as  a 
separate  entry  and  corresponds  to  a  false  alarm  if  classified  as  a  UXO 
target  and  to  a  correct  discrimination  when  classified  as  non-UXO  by  the 
demonstrator. 

c.  The  classification  column  on  the  left  side  of  the  matrix  lists  the  demon¬ 
strator’s  declaration  for  each  detected  UXO  item. 

d.  The  totals  in  the  right-hand  column  of  the  matrix  correspond  to  the  total 
number  of  items  declared  by  the  demonstrator  as  a  particular  class. 

e.  The  “%  Classified”  row  indicates  the  percentage  of  detected  targets,  for  a 
given  target  class,  that  were  correctly  classified. 

/  The  “%  Classified  by  Class”  indicates  the  percentage  of  detected  targets, 
for  the  given  classes  of  projectile,  mortar,  roeket,  and  practice  bomb,  that 
were  correctly  classified. 

g.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  Low/Med 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
ineorrectly  classified  as  nonordnance  with  low  or  medium  confidence. 

h.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  High 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  high  confidence.  This 
classification  error  carried  the  highest  cost  penalty  (equivalent  to  the  cost 
of  a  complete  resurvey  of  the  area). 

Note  that  only  one  demonstrator  classified  their  declarations  with  the  given 
target  classes. 

The  seeond  type  of  classification  matrix  is  the  aggregate  size  classification 
matrix.  This  matrix  includes  the  following  entries: 

a.  The  entry  below  the  demonstrator’s  name  indicates  the  demonstration 
Area. 

b.  The  classes  across  the  top  of  the  matrix  are  the  aggregate  size  elasses  of 
the  items  emplaced.  The  targets  are  grouped  into  three  classes.  The  small 
target  class  contains  the  20-mm  projectile,  the  40-mm  projeetile,  and  the 
BDU  3  practice  bomb.  The  medium  class  contains  the  60-nim  mortar,  the 
81 -mm  mortar,  the  2.25-in.  rocket,  the  2.75-in.  rocket  warhead,  the 
SMAW  rocket,  the  LAAW,  the  BDU-33  practice  bomb,  the  MK-3 
practice  bomb,  and  the  MK-106  practice  bomb.  The  large  class  contains 
the  5-in.  projectile;  the  105-mm  projectile,  the  MK-81  practiee  bomb,  the 
MK-82  practice  bomb,  and  the  MK-83  praetice  bomb.  A  clutter  elass  is 
listed  as  a  separate  entry  and  corresponds  to  a  false  alarm  if  elassified  as 
a  UXO  target  and  to  a  correct  discrimination  when  classified  as  non- 
UXO  by  the  demonstrator. 
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c.  The  classification  column  on  the  left  side  of  the  matrix  lists  the  demon¬ 
strator’s  declaration  for  each  detected  UXO  item.  Again,  the  declarations 
are  grouped  into  three  aggregate  size  classes  as  previously  described  in 
the  target  classes. 

d  The  totals  in  the  right-hand  column  of  the  matrix  correspond  to  the  total 
number  of  items  declared  by  the  demonstrator  as  a  particular  class. 

e.  The  “%  Classified”  row  indicates  the  percentage  of  detected  targets,  for  a 
given  aggregate  size  class,  that  were  correctly  classified. 

f.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  Low/Med 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  low  or  medium  confidence. 

g.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  High 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  demonstrator 
incorrectly  classified  as  nonordnance  with  high  confidence.  Again,  this 
classification  error  carried  the  highest  cost  penalty  (equivalent  to  the  cost 
of  a  complete  resurvey  of  the  area). 

Note  that  only  two  demonstrators  classified  their  declarations  with  enough 
descriptive  information  to  allow  for  the  evaluation  of  aggregate  size 
classification. 

NAEVA  was  the  only  demonstrator  that  submitted  specific  target  identifica¬ 
tions.  Again,  as  with  the  0.5-m  analysis,  their  classification  results,  shown  in 
Figures  Cl  through  C3,  were  very  poor.  Their  results  are  only  slightly  better 
when  aggregated  by  size,  as  shown  in  Figure  C4  for  detections  within  1.0  m. 
Classification  matrices  aggregated  by  size  for  Geophex  are  shown  in  Figure  C5 
for  detections  within  1.0  m.  Geophex  correctly  classified  a  few  more  ordnance  by 
size  than  NAEVA,  but  also  classified  more  non-ordnance  items  as  ordnance. 
NAEVA  and  Geophex  are  the  only  demonstrators  who  attempted  to  identify 
ordnance  by  size. 


ROC-Based  Performance  Assessment 

Assessment  of  detection,  discrimination,  and  false  alarm  rate 
performance  for  1 .0  m 

Figures  C6,  Cl,  and  C8  show  the  demonstrators’  detection  performance  for 
Areas  A,  B,  and  C,  respectively.  The  distance  threshold  for  scoring  a  detection  is 
set  to  1.0  m  for  these  sets  of  plots.  Figure  C6  shows  that  in  Area  A,  all  of  the 
systems  operated  along  very  similar  ROC  curves,  with  the  major  difference  being 
their  selection  of  the  “stop  dig”  or  end  point  threshold.  In  the  cases  of  NRL  and 
Parsons  TM-5  EMU,  the  endpoint  thresholds  were  set  so  high  that  the  operating 
(and  maximum  achievable)  P(det)  was  much  lower  than  those  of  the  other 
demonstrators.  Geophex  achieved  the  best  performance  for  this  area,  but  it  is 
apparent  from  the  steep  slope  of  the  ROC  curve,  that  a  lower  endpoint  threshold 
would  probably  have  resulted  in  increased  P(det)  with  a  relatively  small  increase 

in  total  FAR. 
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Figure  C1 .  NAEVA  EM-63  target  identification  matrix  for  Area  A  within  1 .0  m 
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Figure  C3.  NAEVA  EM-63  target  identification  matrix  for  Area  C  within  1.0  m 
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Figure  C4.  NAEVA  EM-63  size  classification  matrix  within  1 .0  m 
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Figure  C5.  Geophex,  Ltd.  GEM-3  size  classification  matrix  within  1.0  m 
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Figure  C7.  Area  B  -  P(det)  vs.  total  FAR  within  1 .0  m  for  all  demonstrators 
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Figure  C8.  Area  C  -  P(det)  vs.  total  FAR  within  1 .0  m  for  all  demonstrators 
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The  placement  of  the  “stop  dig”  point,  (shown  as  the  “SD”  triangle  in  each  of 
these  figures)  indicates  the  demonstrators’  general  lack  of  confidence  in  their 
discrimination  capability.  With  the  exception  of  NAEVA,  all  other  demonstrators 
placed  their  “stop  dig”  point  at  the  end  of  the  dig  list.  NAEVA’ s  attempt  at 
discrimination  resulted  in  an  operating  P(det)  that  was  10  percent  lower  than  the 
maximum  achievable. 

Figure  Cl  shows  that  all  demonstrators,  with  the  exception  of  GTE,  achieved 
very  similar  ROC-based  performance.  GTE  demonstrated  significantly  poorer 
detection  and  total  FAR  performance  than  the  others.  Overall,  all  of  the  systems 
demonstrated  poorer  ROC-based  performance  in  Area  B  than  Area  A,  confirming 
previous  reports  that  Area  B  contained  significantly  higher  levels  of  geologic 
anomalies  and  metallic  clutter.  As  in  Area  A,  all  demonstrators  with  the  excep¬ 
tion  of  NAEVA,  were  extremely  conservative  in  their  “stop  dig”  point  selection. 

Figure  C8  shows  that  all  of  the  demonstrators  operated  along  very  similar 
ROC  curves.  Again,  all  of  the  threshold  selections  were  very  conservative.  With 
the  exception  of  Geophex  and  NRE,  all  other  demonstrators  performed  better  in 
area  C  than  B,  but  none  performed  better  than  in  Area  A. 

Figures  C9  through  Cl  1  show  the  results  of  reanalyzing  the  data  to  include 
only  targets  larger  than  40  mm  projectiles.  Only  NRE  and  Geophex  performed 
this  analysis  and  resubmitted  revised  dig  lists.  These  figures  show  that  NRE  and 
Geophex  achieve  10  percent  and  7  percent  higher  P(det),  respectively  in  Area  A 
when  the  small  targets  are  ignored.  Considerably  higher  improvements  of 
30  percent  and  15  percent,  respectively,  were  achieved  in  Area  B,  while  only 
15  percent  and  6  percent  improvements  were  achieved  in  Area  C. 

The  small  number  of  submissions,  and  the  lack  of  this  type  of  data  from 
Parsons  precludes  any  comparisons  across  systems  or  with  a  baseline.  In  order  to 
overcome  this  deficiency,  the  demonstrators  dig  lists  that  contained  all  targets 
(including  20  and  40  mm)  were  evaluated  with  the  20/40  mm  targets  removed 
from  the  groundtruth.  No  detection,  false  alarm,  or  missed  target  was  assigned  to 
any  declaration  within  1.0  m  of  these  small  emplaced  targets.  Figures  C12 
through  C14  show  the  results  of  this  evaluation.  These  ROC  curves  show  a  slight 
improvement  in  performance  for  all  demonstrators,  but  no  significant  change  in 
ROC  curve  shape  or  in  the  relative  performance  between  demonstrators.  It  can  be 
concluded  that  all  systems  have  similar  eapability  (or  lack  thereof)  to  detect  the 
smaller  targets  in  the  Kaho’olawe  envirotunent. 

Figures  C15  through  Cl 7  show  the  P(det)  performance  as  a  function  of  Pfp 
for  all  demonstrators,  where  Pfp  is  computed  as  the  ratio  of  the  number  of 
emplaced  clutter  items  included  on  the  dig  list  to  the  number  of  clutter  items 
emplaced.  This  metric  attempted  to  separate  the  effects  of  the  geology  and 
unknown  metal  clutter  from  those  from  known,  emplaced  clutter  items.  These 
ROC  curves  show  very  small  differences  between  demonstrators,  but  most 
importantly,  the  almost  consistently  flat,  diagonal  shape  of  the  curves  indicates 
that  none  of  the  systems  demonstrated  a  capability  to  discriminate  emplaeed 
EIXO  from  emplaeed  metallic  clutter  in  this  environment. 
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Figure  C10.  Area  B  -  P(det)  vs.  total  FAR  within  1.0  m  for  demonstrators  wo  20/40  mm 
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Figure  C12.  Area  A  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  1 .0  m  for  all  demonstrators 
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Figure  C13.  Area  B  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  1 .0  m  for  all  demonstrators 
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Figure  C14.  Area  C  wo  20/40  mm  -  P(det)  vs.  total  FAR  within  1 .0  m  for  all  demonstrators 
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Figure  C17.  Area  C  -  P(det)  vs.  Pfp  within  1.0  m  for  all  demonstrators 
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Figures  C18  through  C20  show  the  systems’  ability  to  diseriminate  ordnanee 
from  nonordnance  (both  geologic  and  metallic)  based  solely  on  signal  strength. 
The  ROC  curves  for  all  demonstrators  tend  to  fall  within  a  narrow  band  and 
support  no  conclusions  regarding  the  various  systems’  discrimination  capability. 


Overall  best  performances 

Overall,  the  best  performances  were  achieved  in  Area  A  where  NAEVA, 
GTL,  Geophex,  Parson  EM  and  flag,  and  Parson  EM-61  digital  had  a  maximum 
achievable  P(det)  of  at  least  75  percent  when  including  detections  within  1 .0  m. 
However,  none  of  the  demonstrators  reached  the  P(det)  required  for  Kaho’olawe 
Tier  II  clearance  (85  percent).  The  P(det)  in  Areas  B  and  C  were  significantly 
lower  for  all  demonstrators. 
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Figure  C18.  Area  A  -  P(disc)  (prioritized  by  signal  strength)  vs.  totai  FAR  within  1 .0  m  for  all 
demonstrators 
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Figure  C19.  Area  B  -  P(disc)  (prioritized  by  signal  strength)  vs.  total  FAR  within  1.0  m  for  all 
demonstrators 
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Figure  C20.  Area  C  -  Pdisc  (prioritized  by  signal  strength)  vs.  total  FAR  within  1 .0  m  for  all 
demonstrators 


C40 


Appendix  C  Evaluation  of  Data  at  1 .0  m 


Appendix  D 

Mean  Location  Error  Analysis 


This  Appendix  contains  the  scatter  graphs  of  the  mean  location  error  in 
northing  and  easting  for  each  emplaced  item,  grouped  by  areas,  shown  in 
Figures  D1  through  D3.  Also,  scatter  graphs  of  the  mean  location  error  in 
northing  and  easting  for  each  demonstrator  are  given  for  each  area  in  Figures  D4 
through  D6.  The  graphs  indicate  sufficient  grouping  of  the  targets  within  0.5  m 
of  their  placement  versus  their  reported  detection  location  by  the  demonstrators. 
Item  2C  623  was  the  only  emplaced  item  that  was  an  ordnance,  having  at  least 
three  demonstrators  with  the  same  positioning  error  over  1.0  m.  The  depth  of  this 
item  was  0.7  m.  The  location  error  probably  resulted  from  the  detection  of  non- 
emplaced  clutter. 

The  following  emplaced  items,  targets  2E  640,  ID  542,  ID  554,  and  ID  550, 
had  a  mean  location  error  over  1 .0  m.  Of  these  targets,  ID  550  was  the  only  one 
that  had  three  or  more  demonstrators  with  the  same  positioning  error. 

The  following  emplaced  items,  targets  3E  71 1,  5D  841, 4C  761,  5C  821,  3E 
718,  3E  715,  5B  793, 4E  777,  5D  842,  5C  823,  3D  97,  and  4C  751,  had  a  mean 
location  error  over  1.0  m.  Of  these  targets,  4C  751  was  the  only  one  that  had 
three  or  more  demonstrators  with  the  same  positioning  error. 

The  following  emplaced  items,  IB  163,  2C  611,  1C  530,  2C  623,  2B  596,  2B 
603,  and  2A  581,  had  a  mean  location  error  over  1.0  m.  Of  these  targets,  2C  623 
was  the  only  one  that  had  three  or  more  demonstrators  with  the  same  positioning 
error. 
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Area  A  Mean  Location  Error  by  Emplaced  Item 
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Figure  D1 .  Area  A  -  Mean  location  error  by  emplaced  it( 


Northing  Error  (m) 


Area  A  Mean  Location  Error  by  Demonstrator 


Easting  Error  (m) 


Figure  D4.  Area  A  -  Mean  location  error  by  demonstrator 
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Figure  D5.  Area  B  -  Mean  location  error  by  demonstrator 
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Area  C  Mean  Location  Error  by  Demonstrator 
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Appendix  E 

Demonstrators’  Data  Collection 
and  Analysis  Plans 


This  Appendix  contains  the  unedited  documents  submitted  by  each  of  the 
demonstrators  as  they  describe  their  planned  or  actual  methodologies  for  per¬ 
forming  the  field  surveys  and  analyze  the  data  as  part  of  the  Kaho’olawe 
demonstrations.  Note  that  Geophysical  Technology  Limited  (GTL)  did  not 
submit  the  required  documents. 
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E.1.  NAEVA  Geophysics  Inc. 


Survey,  Quality  Control  and  Data  Processing 
Methodology 

Advanced  UXO  Discrimination/Discrimination 
Technology  Demonstration 
Kaho’olawe  Island,  Hawaii 

NAEVA  Geophysics,  Inc. 

North  American  Exploration  of  Virginia,  Inc. 

19  November  2001 


Introduction,  NAEVA  Kaho’olawe  Procedures 

After  14  days  of  testing,  calibration,  and  algorithm  development  (for  basalt 
response  discrimination)  on  Kaho’olawe  (September  17  -30,  2001),  North 
American  Exploration  of  Virginia,  Inc.  (NAEVA)  (with  Geophysical  Associates 
(GPA)),  demonstrated  at  Kaho’olawe  during  the  4-day  penod  October  1^,  2001. 
The  overall  setup  for  the  exercise  is  described  in  the  Advanced  UXO 
Discrimination/Discrimination  Technology  Demonstratiori  Plan  (31  Aug.  2001) 
by  the  Naval  Explosive  Ordnance  Disposal  Technolo^  Division 
(NAVEODTECHDIV).  Multiple  samples  of  20  to  30  inert  ordnance  types  and  a 
variety  of  nonordnance  and  clutter  items  were  emplaced  on  each  of  a  1 -hectare 
test  grid  and  ten  30-  x  30-m  test  grids.  The  schedule  and  budget  allowed 
approximately  2  days  per  hectare. 

Measurements  were  also  made  on  a  small  self-evaluation  test  grid  and  in  a 
small  trench.  NAEVA  personnel  respected  all  issues  and  procedures  in  the  Site 
Health  and  Safety  Plan. 


14-Day  Field  Test,  Calibration,  and  Revision-Optimization  Period 

Objectives  during  the  test,  calibration,  and  revision-optimization  period 
were:  (a)  evaluation  of  new  compensation  coil  and  other  noise  reduction 
measures,  and  (b)  measurement  and  evaluation  of  the  spatially  variable  back¬ 
ground  response  of  basalt  and  basaltic  soils  in  various  locations. 

To  these  ends,  NAEVA  made  repeated  measurements  over  basaltic  matenals 
on  the  test  grids  (with  and  without  metallic  targets)  at  various  speeds  (including 
static),  with  and  without  the  compensation  coil,  at  both  medium-  and  high-pulse 
repetition  rates,  and  with  careful  zero  calibration  procedures  to  minimize  instru¬ 
ment  drift.  The  main  objective  during  the  revision  -  optimization  period  was  to 
determine  whether  the  high  background  response  of  basaltic  materials  could  be 
removed  (by  auto-leveling)  and/or  incorporated  into  the  chi-squared  fitting 
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procedures  (by  use  of  a  characteristic  basalt  decay  vector).  The  latter  proved 
more  feasible. 


Background  response  of  basalt  was  spatially  variable  on  the  test  grid. 

Figure  ELI  displays  stacked  profiles  from  a  single  10-m  lane  (test  grid  lane 
0-10),  with  highly  variable  basalt  background  response.  Figure  El. 2  shows 
typical  basalt  amplitude  response  decay  (log-linear). 

Field  and  bench  tests  revealed  that  the  NAEVA  EM-63  instrument  had 
satisfactory  instrument  noise  in  the  medium  frequency  (26  time  gates)  mode. 
However,  the  instrument  drift  rate  was  excessive,  espeeially  in  the  early  gates 
(10  to  20  mV,  generally  negative,  during  the  first  5  to  10  min  of  operation, 
decreasing  thereafter). 

Parsons  and  the  U.S.  Army  Engineer  Research  and  Development  Center 
(ERDC)  very  kindly  made  their  EM-63  data  acquisition  (DAQ)  consoles  avail¬ 
able  for  comparative  tests.  It  was  determined  that  these  consoles  drifted  less  (5  to 
10  mV  over  5  to  10  min)  but  were  much  noisier  in  all  time  gates.  Figure  El. 3 
compares  drift  and  noise  for  these  three  EM-63  consoles.  We  therefore  decided 
to  continue  to  use  the  NAEVA  rental  console. 


Figure  E1 .1 .  Stacked  profiles  of  test  grid  lane  0-gates  1 :3:10 
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Figure  E1.2.  Basalt  EM-63  Amplitude  Response  Decay  (~1/T) 


It  was  therefore  necessary  to  stabilize  (warm  up)  the  EM-63  on  the  test  bench 
for  as  long  as  possible  (10  min  or  so),  and  then  rezero,  before  surveying  each 
10-m  lane.  Then  it  was  necessary  to  place  the  instrument  on  the  test  bench  again 
in  order  to  determine  the  amount  of  drift  during  the  survey.  Unfortunately,  the 
drift  was  still  substantial  and  nonlinear.  This  necessitated  special  steps  in  post¬ 
processing  to  remove  drift  effects  and  variable  basalt  background  response. 

Specific  Demonstration  Survey  Plan  (Data  Acquisition) 

The  EM-63  was  operated  on  a  nonmetallic  test  table  in  static  mode  for 
approximately  10  min  at  the  beginning  and  end  of  each  lane  file  to  zero  the 
instrument  (away  from  possible  background  response)  before  each  survey  period 
and  check  for  calibration  drift  after  each  survey  period.  Figure  El. 4  illustrates 
EM-63  zeroing  in  air  before  grid  lane  surveying.  A  standard  8.89-cm  (3.5-in.) 
iron  calibration  sphere  was  placed  at  zero  depth,  just  north  of  the  north  end  of  the 
first  survey  line  in  each  lane,  to  verify  stable  amplitude  response.  The  initial  line 
in  each  lane  file  was  surveyed  in  southeast  and  northwest  directions  to  verify  data 
repeatability  and  satisfactory  positional  latency  (lag)  corrections. 

EM-63  data  were  acquired  on  all  three  demonstration  grids.  Data  were  mea¬ 
sured  in  narrow  blocks  or  lanes,  approximately  10-m  wide,  over  the  full  north¬ 
west-southeast  extent  of  each  grid  (or  contiguous  group  of  grids).  This  was  done 
because  of  memory  limitations  in  the  EM-63  and  to  avoid  longer-term  zero 
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Figure  E1.3.  Console  comparison  of  drift 
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Figure  E1.4.  Zeroing  of  the  EM-63  before  surveying 


calibration  drift  (data  acquisition  required  approximately  30  to  60  min  per  10-m 
lane).  Figure  E1.5  illustrates  surveying  with  the  EM-63.  Northeast-southwest 
guide  ropes  (parallel  to  prevailing  wind)  were  spaced  2  m  apart  to  ensure  straight 
survey  lines  with  a  0.5-m  line  spacing. 

In  accordance  with  instructions,  the  demonstration  grids  were  surveyed  in 
three  blocks,  which  were  designated  A  (the  northernmost  four  30-  x  30-m  grids), 
B  (the  1-hectare  grid),  and  C  (the  last  six  30-  x  30-m  grids),  as  shown  in  Fig¬ 
ure  El. 6.  Each  lane  was  numbered  in  order  from  southeast  to  northwest  (AO-10, 
AlO-20,  A20-30,  etc.  for  example).  Each  raw  (binary  EM-63-GPS  composite) 
lane  file  contains  approximately  1  to  2  Mb  of  data.  When  necessitated  by  GPS  or 
other  data  problems,  repeat  lane  files  were  measured,  and  named  A10-20b,  etc. 
(This  paper  (E.l)  was  written  and  submitted  by  NAEVA.  This  was  their  record¬ 
keeping  method,  A 10-20,  and  the  repeat  was  A10-20b.)  GPS  positions  were 
acquired  at  a  rate  of  one  per  second,  and  EM-63  readings  were  collected  at  a  rate 
of  five  per  second,  yielding  a  data  density  of  one  reading  approximately  every  10 
to  20  cm. 
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Figure  E1.5.  Traversing  Kaho’olawe  Demonstration  Grid  B  with 
the  EM-63  (Ropes  at  2-m  intervals) 
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Additional  Decay  Curve  Calibration  Measurements 


Additional  (different)  inert  ordnance  samples  were  made  available  at  the 
Kaho’olawc  demonstration  site.  Bench  test  decay  curve  measurements  were 
made  on  these  additional  items  during  the  period  September  17  through  27,  2001, 
as  time  permitted.  Figure  El. 7  illustrates  additional  ordnance  tested  at 
Kaho’olawc. 


Figure  E1 .7.  Ordnance  samples  available  on  Kaho’olawe 

NAEVA  Data  Quality  Control  Plans 

As  previously  mentioned,  the  EM-63  was  static  tested  for  zero  calibration 
and  instrument  (plus  ambient)  noise  at  the  beginning  of  each  survey  lane  fde.  The 
first  line  was  repeated  (bidirectional)  to  verify  amplitude  and  location  repeat¬ 
ability.  As  soon  as  the  file  was  complete,  it  was  checked  for  data  gaps  and/or 
poor  GPS  position  recovery,  and  portions  were  repeated  if  necessary  (generally, 
resulting  from  poor  satellite  availability). 

The  repeatability  of  the  first  line  in  each  grid  lane  file  (and  the  amplitude 
response  of  the  calibration  sphere)  was  also  verified,  and  terrain  noise  was 
inspected.  Spatially  variable  background  response  was  present  in  the  early  time 
gates  at  the  Kaho’olawc  grids  and  was  removed  from  the  field  data  (or  compen¬ 
sated)  before  target  decay  curves  could  be  compared. 

Figure  El. 8  shows  GPS  position  checks  for  a  problematic  lane  file  (Lane 
B 1 8-28,  tree  at  northeast  end);  black  denotes  GPS  first  quality  “fix,”  while  red 
denotes  GPS  second  quality  “float.”  “Floaf’  positions  were  sometimes,  but  not 
always,  usable. 


Data  Processing 

The  basic  EM-63  data  processing  and  analysis  steps  (as  practiced  at  the 
Jefferson  Proving  Ground  “Five”  demonstration  in  year  2000)  arc  as  follows: 

a.  GPS  checks:  GPS  position  integration  (intcipolation,  latency 
corrections). 
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Figure  E1.8.  Check  Ashtech  GPS  positions,  Lane  B18-28 

b.  Auto-Leveling  (all  gates):  Remove  decaying  background  response  and 
calibration  drift  across  all  time  gates.  (Note,  this  was  not  possible  at 
Kaho’olawe.) 

c.  Visual  Inspection  (profiles  and  plan  contour  maps)  and  Editing:  Remove 
bad  data  points,  recognize  data  gaps,  cut  outside  the  grid,  and  split  lines 
for  GEOSOFT.  Repeat  data  acquisition  (DAQ)  if  necessary,  cropping  to 
0.5m  outside  the  grid  boundaries. 

d.  Target  Picking:  Select  all  targets  over  an  appropriate  amplitude  response 
threshold  established  by  yield  curve  or  data  frequency  distribution 
analysis.  At  Kaho’olawe,  it  was  possible  (and  necessary)  to  discriminate 
basaltic  decay  response  before  target  picking.  Harvest  selected  decay 
curves  for  discrimination  analysis. 

e.  Comparison  of  Decay  Curves:  Compute  Chi-Squared  measure  of  misfit 
from  targets  and  bench  calibration  tests  for  expected  ordnance  items. 

/  Prioritization  of  target  list:  Arrange  in  order  of  increasing  chi-squared 
misfit. 

Auto-leveling  and  target-picking  are  usually  the  most  important  and  difficult  data 
processing  steps.  At  Kaho’olawe,  the  combination  of  excessive  instrument  drift 
and  variable  basalt  background  response  made  auto-leveling  impossible.  Details 
of  the  alternative  EM-63  data  processing  and  analysis  steps  as  practiced  at 
Kaho’olawe  are  as  follows: 

To  remove  the  effects  of  the  instrumental  drift  from  the  data  and  allow  for 
variable  basalt  background  decay  response,  a  multistep  procedure  was  used. 

First,  the  average  decay  curve  of  all  points  in  a  window  (within  30  sec  and  20  m) 
was  subtracted  from  each  individual  decay  curve  in  that  window.  These  adjusted 
decay  curves  were  then  fit  with  the  baseline  basalt  decay,  producing  a  value, 
measuring  the  degree  of  misfit.  The  measurements  were  sorted  by  and  the 
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lowest  75  percent  were  reaveraged,  producing  a  nominal  drift  baseline  without 
the  influence  of  measured  anomalies.  This  baseline  was  then  subtracted  from  the 
data  points  in  the  center  10  m  of  the  window  to  produce  the  dedrifted  data  set,  the 
windows  were  advanced  by  10  m,  and  the  procedure  was  repeated. 

This  dedrifting  produced  a  data  set  with  both  positive  and  negative  basalt 
response,  the  former  where  the  basalt  level  was  higher  than  the  local  average,  the 
latter  where  it  was  lower.  Thus,  the  subsequent  fits,  which  include  a  basalt 
component,  allow  that  component  to  be  positive  or  negative. 

The  dedrifted  data  were  fit  with  a  single  basalt  component  first,  producing  a 
X  map  with  peaks  where  the  instrument  response  was  nonbasalt-like.  These 
peaks  were  picked  with  standard  GeoSoft  software  and  were  then  turned  into 
target  lists.  The  discrimination  of  the  targets  was  performed  by  simultaneously 
fitting  both  a  basalt  component  and  components  from  the  ordnance  library  decay 
curves  to  the  decay  curve  of  the  anomaly.  This  produced  a  for  each  type  of 
ordnance  in  the  library.  These  x^  values  were  subtracted  from  the  basalt-only  x^ 
and  these  Ax^  values  (corresponding  to  log-likelihood  ratios  in  a  model  compari¬ 
son  test)  were  used  to  rank  each  target. 


Target  Lists  Provided  to  Environmental  Security  Technology 
Certification  Program  (ESTCP) 

Prioritized  target  lists  were  provided  to  ESTCP  before  leaving  Maui,  as 
required.  It  should  be  recognized,  however,  that  a  relatively  larger  coil  geometry 
such  as  the  1-  X  1-m  EM-63  is  better  for  larger,  deeper  items,  and  not  good  for 
small,  shallow  items  such  as  20-mm  or  thumb-sized  fragmentation.  This  is  a 
virtue,  when  small  fragments  are  to  be  avoided.  There  is  no  single  coil 
configuration  that  is  optimum  for  all  objects  from  20-mm  to  155-mm  and  large 
bombs. 
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E.2.  Geophex  Ltd. 

October  19,  2001 

Naval  EOD  Tech  Div 
Indian  Head,  MD 

Subjeet:  Field  Report  on  Kaho’olawe  Demo 

Attaehed  disk  contains  the  dig  list  in  aeeordance  with  the  format  provided  in 
the  Demonstration  Plan,  dated  August  31, 2001.  Following  brief  descriptions 
apply: 

•  We  employed  10  frequeneies  for  this  GEM-3  survey:  30;  150;  390;  750; 
1,470;  2,970;  5,910;  11,910;  23,850;  47,970Hz. 

•  The  “signal  strength”  is  derived  by  summing  the  differences  in  the 
quadrature  responses  among  all  frequencies.  This  was  found  to  be  most 
immune  to  the  magnetic  geology  of  the  island. 

•  The  signal  strength  was  used  for  ranking  the  list. 

•  A  threshold  signal  strength  was  used  to  derive  the  dig  list  without  the 
25-mm  projeetiles. 

•  The  last  eolumn  (added)  denotes  the  western  comer  of  the  grid  where 
eaeh  target  belongs.  For  instanee,  lA  denotes  the  grid  bounded  by 
Comer  Post  1  A,  IB,  2B,  and  2A. 


We  attempted  with  difficulty  to  classify  and  identify  each  anomaly.  In  an 
attempt  to  minimize  the  library  size  (16  ESTCP  Standard  UXO  and  13 
Kaho’olawe  UXO:  a  total  of  29  UXO),  we  used  the  following  eight  groups.  ES 
stands  for  the  ESTCP  Standard  UXO  and  KH  for  the  Kaho’olawe  UXO  provided 
at  the  QC  Site. 


Group  1 
Group  2 
Group  3 
Group  4 


Group  5 


Group  6 
Group  7 
Group  8 


Small 

KH: 

MT  fuse 

Small 

ES: 

20-mm  HE,  20-mm  proj.  M97,  40-mm  MK2 

Small 

KH: 

BDU-3  submunition 

Medium 

ES: 

57-nim;  60-mm;  2.75-in.  WHD  w/o  Fuse 

KH: 

60-mm  canister,  81 -mm  ill  w/fin,  SMAW 
roeket,  81 -mm  canister 

Medium 

ES: 

2.75-in.  Warhead;  81 -mm  mortar 

KH: 

2.75-in.  rocket,  MK106  practiee  bomb,  2.25-in. 
roeket,  BDU33,  2.75-in.  rocket 

Medium 

KH: 

MK3  Practice  bomb,  LAW  roeket  motor 

Medium 

ES: 

60-mm;  76-mm;  81-nim  mortar 

Large 

ES: 

105-mm  Apers;  4.2-in.  HE;  105-mm  Heat,  5-in. 
proj;  152-mmproj;  155 -mm  proj 
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Thank  you  for  giving  us  this  opportunity  to  participate  in  this  challenging 
and  interesting  technology  demonstration  project. 

Sincerely, 


I.J.  Won 
Geophex,  Ltd. 
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E.3.  Naval  Research  Laboratory 

MTADS  Data  Acquisition  and  Analysis  Operations  at  Kaho’olawe 
Data  Acquisition 

The  Calibration  Site  and  the  30-m  grids  and  1-hectare  grid  on  the  QA  Site 
were  surveyed  with  the  MTADS  man-portable  electromagnetic  (EM)  system;  the 
Calibration  site  was  also  surveyed  with  the  MTADS  man-portable  magnetometer 
system.  The  man-portable  magnetometer  array  supports  2  Geometries  Model  822 
sensors  with  a  horizontal  separation  of  0.25  m  and  a  height  above  ground  of 
0.20  m.  The  EM  platform  used  our  current  high-power  single-gate  version  of  the 
EM  61  with  0.5-  x  1.0-m  transmit  and  receive  coils.  The  EM  sensor  was 
deployed  with  the  long  dimension  cross-track  at  a  transmitter  coil  height  of 
0.20  m. 

The  Calibration  site  was  marked  in  a  North/South  direction  using  twine  on 
1.5-m  spacing  to  define  the  survey  lanes.  Survey  tracks  were  0.5  m  apart.  The 
site  was  surveyed  first  with  the  EM  system  and  then  with  the  magnetometer 
platform. 

The  30-m  grids  and  the  1 -hectare  sites  were  marked  as  a  single  site  in  a 
Northeast/Southwest  direction  using  twine  on  1.5-m  spacing.  Survey  tracks  were 
0.5  m  apart.  The  30-m  grids  were  completed  in  a  single  EM  survey  using  survey 
tracks  stretching  the  entire  length  of  the  site.  The  1 -hectare  site  was  completed  as 
a  single  EM  survey,  beginning  at  the  northwest  boundary. 

The  EM  anomaly  map  for  the  Calibration  Site  is  shown  in  Figure  E3.1.  It  is 
presented  as  an  interpolated  image  on  a  175-mV  scale.  The  bright  features 
stretching  along  the  west  and  east  boundaries  are  returns  from  80d  steel  nails  that 
were  used  to  stake  the  twine.  These  nails  also  served  as  timing  fiduciary  markers 
to  calibrate  the  timing  offsets  for  the  EM  system. 

Figure  E3.2  shows  the  EM  anomaly  Sitemap  for  the  QA  Site,  including  both 
the  30-m  grids  and  the  1 -hectare  area.  This  interpolated  image  is  presented  on  a 
1,000  mV  scale.  About  50  of  the  brightest  targets  are  readily  apparent  on  this 
scale,  as  are  the  80d  steel  nails  at  the  ends  of  the  survey  lines. 

In  all  the  EM  presentations  in  this  document,  and  for  the  purposes  of  our 
analysis,  the  EM  data  were  smoothed  with  a  20-point  “down-the-track”  demedian 
filter.  The  GPS  navigation  data  were  smoothed  with  a  25-point  filter,  and  the  EM 
data  contain  a  timing  correction  of  3  ms.  The  demedian  filter  suppressed  the 
larger-scale  geological  features  without  affecting  the  presentation  or  fitting  for 
most  UXO  targets.  During  the  EM  analysis  process,  a  data  set  was  also  displayed 
that  was  smoothed  with  a  1,000-point  down-the-track  filter.  This  data  set  was 
used  when  fitting  larger  UXO  targets  whose  signatures  were  distorted  by  the 
much  shorter  scale  20-point  filter. 
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Figure  E3.1.  EM  survey  of  the  Kaho’olawe  Calibration  Grid.  The  presentation 
uses  a  20-point  demedian  fiiter  in  an  interpolated  image 

Data  Analysis 

The  Calibration  Site 

Figure  E3.3  shows  the  EM  survey  of  the  southwest  third  of  the  Calibration 
Site  both  as  an  interpolated  image  and  as  a  pixel  plot.  Individual  data  points  arc 
sometimes  apparent  in  the  pixel  plot.  The  overlay  of  the  target  locations  and 
identities  shows  that  most,  but  not  all,  targets  are  apparent  in  either  presentation. 
In  the  Calibration  Site  the  20-point  demedian  filter  effectively  removed  much  of 
the  interfering  geological  return.  Table  E3.1  shows  a  summary  of  our  analysis  of 
the  Calibration  Site,  along  with  the  groundtruth.  The  targets  in  the  ESTCP  Grid 
were  much  more  detectible  than  the  targets  in  the  remainder  of  the  site.  Overall, 
in  the  Calibration  Site  we  concluded  that  43  of  the  87  targets  would  likely  not 
have  been  detected  without  knowledge  of  the  groundtruth.  The  targets  that  would 
likely  be  missed  in  a  blind  EM  survey  arc  shaded  in  the  right  column  in 
Table  E3.1.  Most  of  the  large  deep  targets  (projectiles  or  bombs)  arc  buried 
below  the  detection  limit  of  the  instrument.  The  smaller  targets,  up  to  and  includ¬ 
ing  many  of  the  60-  and  81 -mm  mortars  would  not  be  detected  in  a  blind  survey. 
The  smaller  targets  were  lost  primarily  because  of  the  geological  interference 
structure.  The  deepest  targets  would  probably  be  beyond  our  detection  limit,  even 
without  the  geological. 
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Figure  E3.2.  EM  survey  of  the  QA  Site  at  Kaho’olawe.  The  presentation  is  an 
interpolated  image,  using  a  20-point  demedian  filter.  Note  the 
different  sensitivity  scale  from  Figure  E3.1 


Figure  E3.4  shows  a  presentation,  on  a  similar  area  scale,  of  an  anomaly 
image  from  the  magnetometer  survey.  The  geologieal  interference  is,  of  course, 
much  worse  in  the  magnetometry  data  than  in  the  EM  data.  The  data  shown  in 
Figure  E3.4  are  highly  filtered  (15  point,  down-thc-track  demedian  filter). 
Unfortunately,  we  did  not  have  access  to  more  sophisticated  data  processing 
filters  (or  the  time  to  use  them)  to  improve  the  magnetometer  analysis.  Many  of 
the  targets  arc  apparent  in  the  magnetometer  data,  however,  only  the  deepest 
large  targets  were  dctcctible  in  the  magnetometer  data  but  not  in  the  EM  data.  In 
Figure  E3.4  we  have  circled  dipole  signatures  that  are  probably  associated  with 
the  identified  target  assignments.  The  circled  targets  were  declared  as  not 
dctcctible  in  the  EM  data.  Because  of  time  constraints,  and  because  we  did  not 
have  access  or  time  to  develop  new  data  processing  tools,  we  chose  not  to 
conduct  magnetometry  surveys  of  the  QA  Site. 
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Figure  E3.3.  Southwest  Quadrant  of  the  Calibration  Site,  A  comparison  of  pixel  and  interpolated  image 
presentations  of  the  EM  survey  are  shown 


The  QA  Site 

The  QA  Site,  shown  in  Figure  E3.2,  is  much  more  highly  disturbed  than  the 
Calibration  Site.  Figure  E3.5  shows  a  pixel  presentation  of  part  of  the  IE  30-m 
grid  on  about  the  same  scale  as  the  pixel  plot  in  Figure  E3.3.  The  entire  QA  Site 
looks  similar  when  viewed  on  this  presentation  scale.  There  arc  three  factors 
affecting  the  QA  site  that  make  analysis  more  difficult  for  us  than  on  the 
Calibration  Site.  The  density  of  small  and  intermediate  sized  shrapnel  chunks  is 
much  higher  on  this  site.  In  most  cases,  these  present  0.5-  to  1-m-long  single- 
track  signals,  such  as  dominate  the  image  in  Figure  E3.5.  The  surface  of  the  QA 
site  is  also  much  rougher  than  the  Calibration  Site.  Our  EM  sensors,  unfor¬ 
tunately,  generate  sensor  spikes  when  the  system  bumps  hard  over  a  surface 
feature.  These  signals,  which  one  might  expect  to  be  sensor  data  spikes,  unfor¬ 
tunately  damp  out  over  a  period  of  a  few  tenths  of  a  second,  making  them  almost 
undistinguishablc  from  the  shrapnel  clutter  signal  returns.  Together,  these  two 
effects  arc  responsible  for  most  of  the  red  signal  return  shown  in  Figure  E3.5. 
This  noise-dominated  data  set  required  that  we  had  to  carryout  analysis  at  less 
sensitive  scales,  and  give  up  even  more  of  the  small  targets  than  in  the 
Calibration  Plot.  The  third  effect,  referred  to  above,  is  the  more  intense  geo¬ 
logical  interference  on  this  site.  The  geological  returns  effectively  mitigated  that 
we  analyze  data,  with  the  visual  guidance  of  the  pixel  presentations,  rather  than 
the  combination  of  pixel  and  interpolated  image  presentations  used  at  the 
Calibration  Site. 
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Note:  The  designation  “IPO”  in  the  Comments  coiumn  signals  that  the  target  data  were  selected  from  the  interpolated  image  presentation.  All  target  fitting  algorithms  are  applied  to 
the  uninterpolated  sensor  data  as  shown  in  the  pixel  plots. 
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Figure  E3.4.  Calibration  Site.  Interpolated  image  from  the  magnetometry  survey 
using  a  15-point  demedian  filter 

Our  current  EM  data  analysis  uses  what  we  call  the  three-beta  analysis 
algorithm.  This  provides  an  analysis  that  generates  three  parameters  that  roughly 
eorrelate  with  size  of  the  primary  orthogonal  dimensional  axes  of  the  target.  We 
use  these  beta  parameters  as  elues  to  target  shape  to  provide  information  to  use  in 
classification  decisions.  The  geological  interferences,  and  the  noise  signals  from 
the  shrapnel  clutter  and  sensor  bouncing,  rendered  the  three-beta  parameters 
effeetively  useless  as  analysis  tools.  Therefore,  the  primary  analysis  took  place 
using  the  previously  developed  baseline  EM  analysis  approach.  In  this  approach, 
important  analysis  parameters  are  location,  depth,  and  size.  Shape  information,  to 
the  extent  that  it  is  available,  is  gleaned  from  the  detrend  presentation  in  the 
model  fit  analysis  window.  The  “goodness  of  fit”  parameter  is  used  primarily  as 
an  evaluation  tool  to  guide  boxing  the  data  chip  for  analysis  and  as  a  guide  in 
editing  the  data  seleeted  for  target  fitting. 
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Figure  E3.5.  Pixel  presentation  of  part  of  Grid  1 E  in  the  QA  Site  from  the  EM 
survey.  The  pixel  presentation  uses  the  same  settings  as  the 
Calibration  Site  presentation  shown  in  Figure  E3.3 

Presentations,  such  as  shown  in  Figure  E3.6  were  used  to  guide  the  analysis. 
This  presentation  shows  the  targets  chosen  in  Grid  1C.  Target  numbers  are 
deleted  so  that  the  sensor  data  can  be  seen.  To  reduce  the  importance  of  shrapnel 
returns,  we  were  guided  hy  the  upper  coil  data.  We  particularly  sought  out  targets 
with  returns  in  adjacent  tracks  on  the  assumption  that  larger  targets  would  appear 
on  multiple  tracks.  Some  single-track  targets  were  still  reported.  We  typically 
rescaled  the  presentation  many  times  to  seek  out  smaller  targets  and  to  try  to 
evaluate  geological  returns.  However,  because  of  the  noise  returns  (such  as 
shown  in  Figure  E3.5)  we  were  limited  to  working  with  fairly  high  intensity 
signals.  We  were  aware  that  this  mitigated  against  our  detection  of  both  small 
targets  and  the  deep  targets  that  had  smaller  intensity  signal  returns. 
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Figure  E3.6.  QA  Site,  Grid  1C.  Pixel  presentation  of  the  EM  survey  showing 
targets  selected  for  analysis 
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1.0  Data  Collection  Methods 


1.1  Geonics  EM-61  Field  Discrimination 

The  first  method  employed  by  Parsons  Infrastrueture  and  Teehnology 
(Parsons)  will  be  a  field  discrimination  (“mag  &  flag”)  survey  utilizing  a 
standard  Geonics  EM-61  (EM-61)  full  meter  coil  (1  m  by  1  m). 


1.2  Geophysical  Technology  Limited  TM-5EMU  Field  Discrimination 

The  second  method  employed  by  Parsons  will  be  a  field  discrimination 
(“mag  &  flag”)  survey  utilizing  a  standard  Geophysical  Technology  Limited 
(GTL)  TM-5EMU  (EMU). 


1.3  Geonics  EM-61  Postprocessed  GPS-Integrated  Digital  Data 

The  third  method  employed  by  Parsons  will  be  a  survey  utilizing  a  Geonics 
EM-61  postprocessed  GPS-integrated  digital  data  system.  This  method  will  be 
employed  only  if  time  constraints  allow  (i.e.,  the  first  two  methods  have  been 
completed  with  enough  time  remaining  to  allow  digital  data  collection). 

Surveys  to  be  independent  of  each  other 

All  three  surveys  will  be  conducted  independently  of  each  other  utilizing 
separate  teams. 


2.0  Survey  Procedures 

2.1  Lane  Spacing 

EM-61  and  EM-61  Digital  Data:  For  the  EM-61  field  discrimination  and 
postprocessed  GPS-integrated  digital  data  surveys,  a  1-m  lane  spacing  will  be 
utilized.  The  EM-61  coil  is  1-m  wide,  which  allows  a  1.5-  to  2-m-wide  effective 
survey  path  (dependent  on  target  depth,  orientation,  composition,  and  size).  The 
overlap  will  ensure  that  sufficient  coverage  of  the  test  grids  is  obtained. 

EMU  Lane  Spacing:  For  the  EMU  field  discrimination  survey,  a  1-m  lane 
spacing  will  be  utilized.  The  standard  EMU  has  a  0.5-m-diameter  coil.  The 
sweep  range  (lateral  arc)  of  our  operators  ranges  between  1.5  and  2  m.  The 
overlap  created  by  the  sweep  range  in  conjunction  with  the  1-m  lane  path  will 
ensure  that  sufficient  coverage  of  the  test  grid  is  obtained. 


Appendix  E  Demonstrators’  Data  Collection  and  Analysis  Plans 


E25 


2.2  Sampling  Rates 

EM-61  and  EM-61  Digital  Data:  For  the  EM-61  field  discrimination  and 
postprocessed  GPS-integrated  digital  data  surveys,  the  EM-61  will  be  set  in  the 
“Auto  Mode,  Extra  Fast  Mode.”  Extra  fast  mode  results  in  7  readings  per  second. 
The  Ashtech  Reliance  GPS  system  records  GPS  positional  readings  at  a 
minimum  rate  of  30  readings  per  minute. 

EMU:  The  EMU  takes  58  samples  per  second. 


2.3  Accuracy  of  Navigation  and  Tracking  System 

The  navigation  system  used  during  collection  for  all  three  survey  methods 
will  be  a  visual  system  utilizing  cones  and  tapes.  Tapes  will  be  placed  approx¬ 
imately  every  30  m  perpendicular  to  the  travel  direction.  Cones  will  be  placed  on 
each  tape  at  the  current  lane.  Given  the  relative  flatness  of  the  terrain  comprising 
the  test  grids,  this  navigational  system  will  provide  less  than  0.5  m  of  lateral 
offset  from  the  intended  line  path.  During  field  discrimination,  each  anomaly  will 
be  investigated  in  a  perpendicular  direction  to  the  travel  lane,  which  will  result  in 
a  radial  error  of  less  than  0.5  m. 

An  Ashtech  Z-Surveyor  GPS  system  will  be  utilized  to  record  the  position  of 
each  anomaly  detected  in  both  the  EM-61  field  discrimination  survey  and  the 
TM-5  EMU  field  discrimination  survey.  The  real-time  accuracy  of  this  system  is 
±0.01  m. 

An  Ashtech  Reliance  GPS  system  will  be  utilized  for  the  EM-61  digital  data 
survey.  The  postprocessed  accuracy  of  this  system  is  ±0.1  m. 


2.4  Data  Format 

Data  will  be  submitted  in  an  ASCII  format  that  includes  sensor  location  data 
in  UTM  coordinates,  followed  by  sensor  readings.  Header  information  required 
under  Section  6.1  of  the  Technology  Demonstration  Plan  will  also  be  included  in 
the  data  files. 


3.0  Data  Collection  Plan 

Parsons  plans  to  collect  the  EM-61  field  discrimination  and  the  EMU  field 
discrimination  simultaneously.  One  team  will  begin  collecting  data  in  the  30-m 
by  30-m  grids  and  a  second  team  will  concurrently  begin  collecting  data  in  the 
one-hectare  grid.  Real-time  surveyor(s)  will  simultaneously  survey  anomaly 
locations.  Parsons  will  not  be  seeking  groundtruth  for  the  first  four  30-m  by  30-m 
grids  prior  to  completing  the  surveys. 

Following  completion  of  the  field  discrimination  surveys,  the  EM-61  digital 
data  collection  will  commence. 
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4.0  Analysis  and  Decision  Process 

4.1  EM-61  Field  Discrimination 

Trained  and  certified  EM-61  operators  will  analyze  the  tonal  and  numeric 
response  of  the  EM-61  in  real-time  along  1-m-wide  lanes.  Each  5-mV  rise  above 
the  local  background  reading  will  be  investigated  as  a  possible  anomaly.  The 
operator  will  investigate  each  potential  anomaly  perpendicular  to  the  lane  path  as 
well  as  along  the  path.  Anomalies  will  be  identified  based  on  the  amplitude, 
slope,  wavelength,  and  shape  of  the  EM-61  response  in  perpendicular  directions. 


4.2  EMU  Field  Discrimination 

Trained  and  certified  EMU  operators  will  analyze  the  tonal  characteristics, 
graphical  characteristics,  and  numeric  response  of  the  instrument  in  real  time 
along  1-m-wide  lanes.  Each  EMU  response  above  the  local  background  reading 
will  be  investigated  as  a  possible  anomaly.  Anomalies  will  be  identified  based  on 
the  amplitude,  slope,  wavelength,  shape,  and  size  of  the  EMU  response. 


4.3  Geonics  EM-61  Postprocessed  GPS  Integrated  Digital  Data 

Trained  and  certified  EM-61  operators  will  utilize  an  EM-61  and  an  Ashtech 
Reliance  GPS  receiver  to  simultaneously  record  digital  EM61  response  and  GPS 
positional  data  along  1-m-wide  lanes.  The  EM-61  data  and  postprocessed  GPS 
positional  data  will  be  synchronized  through  their  recorded  time  stamps.  These 
data  will  be  downloaded  and  analyzed  offsite  by  a  trained  and  certified  data 
analyst.  The  GPS  data  will  be  postprocessed  using  GPS  data  from  an  on-site  base 
station  and  merged  with  the  EM-61  data.  The  resulting  files  (containing  easting, 
northing,  and  electromagnetic  response)  will  then  be  analyzed  using  Geosoft’s 
Oasis  Montaj  software.  The  data  analyst  will  apply  a  lag  correction  to  the  data 
based  on  the  analysis  of  a  field-conducted  daily  instrument  test.  The  analyst  will 
create  profiles  of  the  EM-61  response  along  each  lane  for  both  the  top  coil  and 
the  bottom  coil.  Grid  maps  will  be  produced  using  the  raw  EM-61  data  as  well  as 
data  run  through  high  pass,  vertical  derivative,  and  analytic  signal  filters.  The 
data  analyst  will  investigate  lane  profiles,  2-dimensional  maps  and  3-dimensional 
maps  to  identify  anomalies  based  on  a  minimum  5-mV  EM-61  response  above 
the  background,  in  conjunction  with  the  slope,  wavelength,  and  shape  of  the 
response  from  each  coil. 


Appendix  E  Demonstrators’  Data  Collection  and  Analysis  Plans 


E27 


5.0  Personnel  Information 

EM-61  Field  Discrimination 


Type  of  Work 

Personnel 

Field  Discrimination 

Richard  Arts 

Field  Discrimination 

Addie  Johansen 

GPS  Surveying 

Tom  Dickson 

EMU  Field  Discrimination 


Type  of  Work 

Personnel 

Field  Discrimination 

Jon  Lincoln 

Field  Discrimination 

Michael  Ault 

GPS  Surveying 

Kamalei  Hill 

EM-61  Digital  Data  Collection  and  Processing 


Type  of  Work 

Personnel 

Data  Collection 

Dean  Tokishi 

Data  Collection 

Steve  Vuich 

Processing  Geophysicist 

Andy  Gascho 
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